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1 . INTRODUCTION 


This  semiannual  technical  report  covers  research  carried  out  by  the 
Advanced  Teleprocessing  Systems  group  at  UCLA  under  ARPA  Contract  DAHC  \i 


! ' 15-73-C-0368  during  the  period  January  1,  1977  to  June  30,  1977.  Advance-  |j 

[.  ments  have  been  made  on  all  four  contracted  tasks;  namely,  ground  radio  H 

packet  switching,  satellite  studies,  resource  sharing  and  security.  In  jj 

the  following  paragraphs,  we  describe  the  progress  and  point  to  the  list 
of  references  which  represent  the  published  work  resulting  from  this  i ; 

supported  research.  ;; 


Following  this  summary  is  a list  of  publications  produced  as  a result 
of  the  recent  research  on  this  contract  covering  the  six  months  being 
reported.  This  list  contains  only  those  articles  and  reports  which,  in 
fact,  did  appear  in  print.  Papers  which  have  been  submitted  (of  which 
there  are  many)  are  not  listed  here,  but  will  be  listed  in  future  reports 
as  they  appear  in  the  published  literature.  As  usual,  we  devote  the  main 
body  of  this  report  to  the  detailed  presentation  of  one  aspect  of  this  over- 
all research,  and  we  simply  mention  the  other  areas  briefly  in  this  summary. 

The  research  reported  in  the  main  body  of  this  document  discusses  the 
effect  that  packet -switching  networks  have  on  "stream  traffic".  An  example 
of  stream  traffic  is  packetized  speech;  other  examples  are  real-time 
traffic  generated,  for  example,  from  real-time  sources  such  as  television 
cameras,  temperature-monitoring  systems,  etc.  The  characteristic  of  this 
traffic  is  that  it  has  a time  constraint  as  well  as  a throughput  constraint. 
The  time  constraint  is  that  the  roundtrip  delay  must  not  exceed  a threshold 
before  the  usefulness  of  the  data  becomes  lost;  in  addition,  the  timing  be- 
tween delivered  packets  must  not  be  too  large  or  the  continuity  of  the 
signal  is  lost.  The  throughput  constraint  comes  about  because  the  traffic 
continues  to  be  generated  and  requires  a moderate  (if  not  high)  bandwidth 
through  the  network.  However,  an  advantage  of  stream  traffic  is  that  not 
all  of  it  must  be  delivered;  in  some  sense,  it  is  volatile  and  becomes  stale. 
This  kind  of  traffic  is  very  different  from  that  which  the  ARPANET  was 
originally  designed  to  carry  and,  as  one  might  expect,  some  unusual  phenomena 
occur  when  sent  through  a packet-switched  network.  The  particular  details 
in  the  following  report  represent  the  Ph.D.  dissertation  of  William  E.  Naylor 
(chairman,  Leonard  Kleinrock)  and  the  work  is  entitled  "Stream  Traffic  Com- 
munication in  Packet-Switched  Networks."  Whereas  the  effort  was  to  study 
stream  traffic  flow  through  a network,  it  turns  out  that  a large  effort  was 
devoted  to  studying  the  routing  procedure  (in  particular,  the  periodic  up- 
dating procedure)  in  the  ARPANET.  It  was  found  that  this  routing  procedure 


severely  affected  and  degraded  stream  traffic  performance  and  so  a number 
of  suggestions  were  made  and  evaluated  as  regards  the  priority  ordering 
of  updating.  A loop-free  algorithm  for  updating  routing  tables  is  also 
presented  and  proven  to  be  loop-free.  Lastly,  the  tradeoff  between  gaps 
in  transmission  and  delay  in  reception  of  that  transmission  is  studied. 

A number  of  simulations  and  measurement  experiments  are  reported  which 
confirm  and  support  the  analytical  treatment  given  in  this  work.  Further 
details  can  be  found  in  the  abstract  and  dissertation  reproduced  in  the 
main  body  following  the  list  of  publications. 

The  first  four  references  in  the  list  of  publications  pertain  to  the 
behavior  and  analysis  of  landbased  computer  networks.  The  first  paper 
describes  some  additional  experimentation  for  the  design  of  packet  networks 
and  summarizes  some  of  the  successful  design  procedures  which  we  have  de- 
veloped. The  second  paper  on  the  analysis  of  buffer  allocation  schemes 
describes  various  ways  for  sharing  storage  in  a multiplexing  node  within  a 
computer  network.  The  third  paper  summarizes  a portion  of  a study  of  hier- 
archical routing  in  large  networks  and  shows  the  tradeoff  between  the  length 
of  routing  tables  and  the  path  length  presented  by  those  routes.  The  fourth 
paper  has  to  do  with  flow  control  and  throughput  in  the  ARPANET.  It  points 
out  some  traps  and  degradations  which  may  arise  in  networks,  shows  how  to 
correct  them  and  then  warns  against  the  kinds  of  effects  which  produce  dead- 
locks. The  fifth  paper  on  packet  switching  in  radio  channels  represents  a 
contribution  to  our  work  in  ground  radio  packet  switching.  It  presents  a 
new  scheme  known  as  mini-slotted  alternating  priority  (MSAP)  and  presents 
the  performance  profile  for  this  system.  Its  particular  usefulness  is  in 
the  heavy  traffic  case  since  it  achieves  a capacity  of  1 and  still  provides 
rather  efficient  communications  delays.  Further,  it  discusses  the  regions 
in  which  various  access  schemes  are  optimal.  Overall,  the  advancements  in 
ground  radio  packet  switching  has  moved  alone  extremely  well.  A number  of 
publications  have  been  accepted  and  will  be  reported  on  in  the  next  semiannual 
technical  report. 

The  main  report  on  stream  traffic  in  packet  networks  is  eiven  after 
the  following  list  of  publications. 


List  of  Publications 


Gerla,  M.  and  L.  Kleinrock,  "On  the  Topological  Design  of  Distributed 
Computer  Networks,"  IEEE  Transactions  on  Communications,  Vol.  COM-25, 

No.  1,  January  1977,  pp.  48-60. 

Kermani,  P.  and  L.  Kleinrock,  "Analysis  of  Buffer  Allocation  Schemes 
in  a Multiplexing  Node,"  Conference  Record,  International  Conference 
on  Communications,  Vol.  2,  Chicago,  Illinois,  June  1977,  pp.  30.4-269 
to  275. 

Kleinrock,  L.  and  F.  Kamoun,  "Hierarchical  Routing  for  Large  Networks: 
Performance  Evaluation  and  Optimization,"  Computer  Networks,  Vol.  1,  No.  3, 
January  1977,  pp.  155-174. 

Kleinrock,  L.  and  H.  Opderbeck,  "Throughput  in  the  ARPANET--Protocols  and 
Measurement,"  IEEE  Transactions  on  Communications,  Vol.  COM-25,  No.  1, 
January  1977,  pp.  95-104 

Kleinrock,  L.  and  M.  Scholl,  "Packet  Switching  in  Radio  Channels:  New 
Conflict-Free  Multiple  Access  Schemes  for  a Small  Number  of  Data  Users." 
Conference  Record,  International  Conference  on  Communications,  Vol.  2, 
Chicago,  Illinois,  June  1977,  pp.  22.1-105  to  111. 
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ABSTRACT 


This  research  deals  with  the  transmission  of  stream 
traffic  through  packet  switched  networks.  Stream  traffic 
communication  is  characterized  by  1)  a requirement  of  small 
response  time  and  moderate  throughputs  the  fact  that  2)  tim- 
ing is  an  integral  part  of  the  information  and  that  3)  the 
information  is  redundant  and  somewhat  tolerable  to  loss. 
Remote  voice  communication  provides  an  example  of  stream 
traffic  communication.  Traditional  dedicated  communication 
systems  supporting  stream  traffic  have  exhibited  fixed  capa- 
city and  fixed  delay.  In  such  systems  each  user  is  assigned 
a communication  link  whose  capacity  is  large  enough  to  sup- 
port that  user’s  peak  load.  while  timing  is  easily 
preserved  by  such  systems^  it  is  difficult  to  share  communi- 
cation Links.  This  research  explores  systems  in  which  the 
communi ca t i on  channels  are  shared  among  many  users/  thus 
causing  delay  and  capacity  to  vary..^ 

The  first  area  of  concentration  is  an  examination  of 
some  important  factors  which  adversely  affect  delay  in  pack- 
et switched  networks.  The  ARPANET  is  used  as  an  example  of 
such  a network.  We  focus  on  loop  control  in  adaptive  rout- 
ing/ priority  assignment/  and  the  effect  of  periodic  update 
routing  in  large  networks.  Suggestions  are  offered  for  per- 
formance improvement  in  these  areas.  Analysis  and  simula- 
tion are  used  to  predict  the  magnitude  of  improvement.  "y 
modifying  the  ARPANET  procedures  in  the  manner  suggested/ 


nteasu renent  and  simulation  indicate  that  a 40?:  to  50%  reduc- 
tion in  average  delay  may  be  achieved. 

rven  with  improved  performance  a packet  switched  net- 
work exhibits  variable  delay  due  in  general  to  the  possible 
queueing  of  packets  at  channels  in  the  network.  with  stream 
traffic  it  is  important  to  preserve  the  relative  timing  of 
the  information  as  closely  as  possible.  This  is  cccom- 
plished  by  smoothing  the  departure  of  information  with 
buffering.  This  is  the  second  main  area  of  concentration. 
We  propose  buffering  schemes  which  adapt  to  changing  network 
delay  and  which  trade  output  smoothness  against  buffering 
delay.  Th  ormance  of  the  buffering  strategies  is  com- 


pared by 


IS  and  simulation. 
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CHAPTER  1 
INTRODUCTION 


1.1  R£££ni  bistficy  Qi  ca£k£i  sMllsblos 

liince  the  development  of  the  ARPANET  CRobe  703  in  the 
late  1960s  by  the  Advanced  Research  Projects  Agency  of  the 
United  States  Department  of  Defense^  there  has  been  an  ever 
increasing  amount  of  activity  in  the  area  of  packet  switch- 
ing research/  development/  and  implementation.  The  ARPANET/ 
built  primarily  by  Bolt/  Beranek  and  Newman/  Inc./  Cam- 
bridge/ Massachusetts  (BBN)/  is  the  most  widely  known  exam- 
ple of  a packet  switched  network.  Other  networks  are  emerg- 
ing as  many  agencies/  countries  and  federations  are  current- 
ly funding  research  and/or  building  packet  switched  net- 
works. Among  these  are  WUNCCS  CBeno  713/  NPL  CBarb  703/ 
CYCLADES  CPouz  733/  DATAPAC  CDPAC  743/  C0ST11  [Barb  723/  to 
name  a few.  The  first  commercial  packet  switching  service 
is  now  in  operation  in  the  U.  S.  (operated  by  Telenet  Com- 
munications Corporation)  [Math  753. 

Extensions  to  the  ARPANET  form  of  packet  switching  have 
taken  place  and  have  lead  to  experimental  systems  --  ALOHA 
CAbra  703/  PRNET  Ckahn  753/  SATNET  CKlei  733/  ETHERNET 
CMetc  773.  Currently/  the  interconnection  of  such  networks 
CNcKe  74a3  and  the  standardization  of  protocols  CPouz  753/ 
CHove  763  are  each  of  considerable  interest  in  the  data 
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communications  community-  Also  of  interest  are  aspects  of 
secure  communications  CFarb  7S] . 

Iti£  &aii£Dt  fsaiunsi  gf  cisisei  aaiubioa 

Here  we  describe  the  important  (so  far  as  we  are  con- 
cerned here)  properties  of  a packet  switched  network.  A 
packet  switched  network  (of  the  ARPANET  form)  incorporates 
switching  computers  within  the  communications  media.  The 
network  consists  of  HOSTs  (i.e.r  packet  or  message  sources 
--  computers  or  terminals)  and  a subnet  to  which  the  HOSTs 
are  attached.  The  subnet  contains  switching  nodes  (called 
IMPS  in  the  ARPANET  CHear  70D)  which  are  connected/  with 
some  topology/  by  a set  of  communications  lines  or  channels. 
HOSTs  are  connected  to  the  network  at  switching  nodes.  » 
source  HOST  sends  a message  (to  a destination  HOST)  by 
delivering  the  message/  with  the  address  of  the  destination 
HOST/  to  its  connected  switching  node.  The  switching  node 
then  breaks  the  message  into  one  or  more  packets.  Each 
packet  is  then  forwarded  (if  necessary)  by  the  switching 
node  to  one  of  its  neighbors  (i.e./  another  switching  node 
which  is  directly  connected  to  the  fcrwardinq  node)  the 
choice  of  neighbors  being  governed  by  the  routing  procedure. 
Each  node/  which  encounters  the  packet/  forwards  it  to  a 
neighbor  the  choice  of  which  is  again  based  on  the  routing 
procedure . 

Often  there  is  some  form  of  error  detection  and 
recovery  in  the  forwarding  process  in  order  to  assure  the 
correct  receipt  of  packets.  In  the  ARPANET/  for  example/ 
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there  is  a checksum  appended  to  the  packet  at  the  sender 
which  is  checked  by  the  receiver.  Only  when  the  checksum  is 
correct  and  buffer  space  is  available  does  the  receiver 
return  (sometimes  "pi qgybacked”  on  reverse  traffic)  an  ack- 
nowledgement (ACK)  to  the  sender.  If  a sent  packet  has  not 
received  its  ACK  within  a certain  time  (currently  128  mi  l- 
liseconds)/  then  the  packet  is  retransmitted. 

The  forwarding  process  is  repeated  until  the  packet 
arrives  at  a switching  node  which  is  connected  to  the  desti- 
nation HOST/  at  which  point  the  packet  is  reassembled  with 
the  other  packets  of  the  same  message.  When  all  its  packets 
have  arrived  the  message  is  sent  to  the  destination  HOST. 
In  the  ARPANET  this  final  step  creates  an  end-to-end  ack- 
nowledgement called  a Ready-for-Next-Message  (RFN»*)  which  is 
sent  back  to  the  sender. 

At  each  forwarding  step/  often  called  a hop/  along  the 
way  a packet  incurs  some  processing  delay  which  is  required 
to  make  the  routing  decision  (i.e./  to  which  neighbor  to 
route  this  packet).  Also/  it  may  encounter  a oueue  of  pack- 
ets waiting  to  be  sent  to  the  neighbor  and  must  therefore 
wait  until  all  (higher  or  equal  priority)  packets  have  been 
served  (i.e./  sent  to  the  neighbor)  before  it  may  use  the 
channel.  In  particular/  this  may  result  in  variable  delay. 
That  is/  a message  sent  from  A to  B at  time  tO  may  experi- 
ence a different  delay  than  a similar  message  sent  at  time 
t1.  The  maximum  attainable  throughput  from  a to  P may  vary 
with  time  as  well/  since  it  will  depend  on  the  level  of 
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interference  from  other  communicating  HOST  pairs. 

Packet  switching  promises  greater  efficiency  of 
resources  by  sharing  those  resources  among  many  usersf  each 
of  which  uses  only  a small  portion  of  the  total  capacity. 
The  savings^  at  least  in  part^  are  passed  along  to  the  user. 
With  the  rapid  growth  of  data  communications  we  shall  surely 
see  more  and  bigger  packet  switched  networks  emerge. 

Because  of  its  digital  nature^  today's  packet  switching 
technology  has  been  designed  and  used  chiefly  for  data  com- 
munications (e.g.^  ter m ina I- t o- comput er  and  comouter-to- 
computer) . Other  forms  of  communication  (i.e.^  voice)^ 
which  have  traditionally  been  accomplished  by  analog 
methods/  are  now  beginning  to  use  digital  technology. 

1.2  )iQi££  9S££  digiui 

The  advent  of  inexpensive/  highly  reliable  digital 
transmission  equipment  has  already  led  to  the  use  of  such 
circuits  in  "short-haul"  voice  communications  CJame  72D. 
Also  some  4C  million  circuit  miles  of  digital  trunks  were  in 
place  by  the  end  of  1975  CFalk  77].  There  seems  little 
doubt  that  the  use  of  digital  transmission  will  increase  as 
time  progresses.  We  are  told  by  Gallager  CGall  77a]  that 
the  military  will  use  an  ati.  digital  telephone  network  in 
the  1980's.  Cost  will  be  the  most  important  reason  for 
conversion  to  digital  transmission.  According  to  Falk 
CFalk  77]  "Bell  Canada  plans  to  save  $40  million  a year  in 
capital  costs  by  using  new  digital  equipment".  Another 
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consideration  is  that  a diqital  sional  is  much  easier  to 
encrypt  than  is  an  analog  signal.  This  could  lead  to  a far 
more  secure  means  of  voice  communication. 

Simple  digitization  requires  a data  rate  of  between  50 
and  60  kilobits  per  second  (kbps)  to  provide  quality  equal 
to  conventional  telephone  equipment.  This  is  extremely  high 
compared  to  current  requirements  in  data  communications. 
Fortunately^  speech  is  highly  redundant  and  can  therefore  be 
significantly  compressed. 

I *3  ^B£S£b  cgmcrg^iiSQ  rggggcgh 

There  are  two  methods  of  compression  currently  under 
study  by  the  ARPA  Network  Speech  Compression  (NSC)  group  - 
(a)  continuous  variable  slope  delta  modulation  (CVSD) 
CForg  74D  and  (b)  linear  predictive  coding  (LPC)  CAtal  713. 
Experimental  studies  of  remote  voice  communication  using 
these  two  schemes  have  already  taken  place  in  the  ARPANET. 
The  CVSD  scheme  transmits  one  bit  per  sample^  whose  value  (1 
or  0)  depends  on  whether  the  last  synthesized  point  is  above 
or  below  the  current  input.  This  scheme  provides  acceptable 
quality  in  the  8 to  20  kbps  region  CForg  743.  The  experi- 
ments in  the  ARPANET  with  this  technique  used  a peak  rate  of 
10  kbps  and  experienced  delays  on  the  order  of  2 to  4 
seconds  between  speaker  and  listener  (over  a 10  hop  network 
path)^  as  reported  by  Cohen  in  CNayl  74a3.  The  LPC  method 
is  based  on  a model  of  the  vocal  tract  and  assumes  that  a 
speech  sample  may  be  approximated  by  a linear  combination  of 
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the  previous  n samples.  The  coefficients  in  the  linear  com- 
bination are  slowly  varying  quantities.  This  algorithm  pro- 
vides high  Quality  speech  in  the  range  of  2.4  to  9.6  kbps 
CAtal  713.  With  refinements  such  as  OELCO  CMaqi  733^  one 
can  achieve  average  data  rates  of  1.2  to  4.8  kbps.  A recent 
experiment  in  the  ARPANET  had  a peak  data  rate  of  4.1  kbps 
and  an  average  of  1.4  kbps/  with  acceptable  quality^  using 
the  LPC  compression  scheme  CMcCa  753. 

This  research  has  created^  in  essence^  a new  form  of 
"data”  communication^  and  provides  the  opportunity  to  apply 
packet  switching  technology  to  the  area  of  remote  voice  com- 
munication. Human  speech  contains  a great  deal  of  redundan- 
cy and  silence.  During  silence/  with  most  dedicated  or 
switched  circuit  systems  the  channel  remains  unused.  Packet 
switching  would  allow  this  dead  time  to  be  used  by  others 
thus  allowing  greater  channel  efficiency  which  eventually 
translates  to  cost  savings. 

1*4  Iii£  &hg£a£t££l£tl££  cl  st££am  £££lll£ 

This  dissertation  investigates  a set  of  problems  relat- 
ed to  the  use  of  packet  switching  technology  for  the  purpose 
of  stream  traffic  communication.  Remote  voice  communication 
is  an  example  of  stream  traffic  communication.  Let  us  de- 
fine stream  traffic.  Stream  traffic  is  characterized  by  the 
following  three  properties: 

1)  small  response  time  and  moderate  throughput  are 
r equi red/ 
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2)  timing  is  an  integral  part  of  the  information#  and 

3)  the  information  contains  redundancy. 

Property  1 allous  for  the  possibility  of  real-time 
interactive  communication  between  two  or  more  locations. 

This  property  alone  makes  stream  traffic  distinguishable 
from  the  two  classical  forms  of  data  communication.  Packet 
switched  networks  have#  in  general#  been  designed  to  carry 
traffic  which  has  traditionally  been  classified  into  two 
categories:  (a)  LD  - low  delay  (interactive)#  and  (b)  HT 
high  throughput  (file  transfer).  As  noted  by  Cohen#  Cpder- 
beck  and  Kleinrock  COpde  74D#  stream  traffic  communication 
falls  into  yet  another  category  (c)  ST  - stream  traffic# 
requiring  both  low  delay  and  moderate  throughput. 

Not  only  are  the  transmission  requirements  of  stream 
traffic  unique#  but  the  information  itself  is  of  a somewhat 
different  nature  than  the  usual  data  communication.  Proper- 
ty 2 indicates  that  each  unit  (bit#  if  you  will)  of  informa- 
tion has  an  associated  (possibly  implied)  time  stamp  and 
that  the  relative  timing  of  the  information  should  be 
preserved  as  well  as  possible  by  the  transmission  media. 

Although  sequencino  is  important#  in  ordinary  data  communi- 
cations there  is  no  notion  of  timing  associated  with  the 
information.  Unlike  traditional  data  communication#  the 
information  in  stream  traffic  is  somewhat  redundant.  This 
means  that  the  information  is  less  vulnerable  to  loss  within  I 

the  system  than  for  traditional  data  communication.  The  ^ 


communication  media  may  lose  a small  fraction  of  the 


information  without  seriously  affecting  the  quality.  This 
is  clearly  distinct  from#  say#  remote  job  entry  in  which  a 
very  precise  specification  in  the  form  of  a program  is  the 
object  of  communication#  and  therefore  no  loss  at  all  can  be 
tolerated . 

This  characterization  of  stream  traffic  is  intended  to 
fit  two-  (or  more)  way  interactive  voice  communication. 
There  are  other  areas  of  communication  which  may  possess 
these  properties  as  well.  Distribution  and  local  broadcast 
of  live  (or  delayed  and  transmitted  from  the  source  in 
real-time)  radio  or  television  programming  would  appear  to 
have  the  three  properties.  While  there  is  no  interaction 
required  of  such  one-way  systems#  tow  delay  may  be  required 
due  to  lack  of  buffer  space  at  the  receiver.  Television  in 
particular  requires  extremely  high  throughput  by  ARPANET 
standards.  Therefore  the  capacity  of  a network  designed  to 
carry  television  must  have  a capacity  so  that  the  television 
transmission  consumes  only  a moderate  portion  of  that  capa- 
city. 

1*5  SiaiggieQi  gf  t.h£  CCfibiSQ! 

There  are  two  main  areas  of  emphasis  which  are  explored 
in  this  dissertation. 

1)  Identify  and  examine  those  design  considerations  (of 
packet  switched  networks)  which  have  a significant 
impact  on  the  performance  of  stream  traffic  communi- 
cation in  a packet  switched  environment. 
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2)  Given  a (nacket  switched)  network/  examine  some 
sending  and  receiving  policies  which  attempt  to  bal- 
ance the  preservation  of  the  relative  timing  against 
end-to-end  delay. 

since  the  ARPANET  is  a convenient  example  of  a packet 
switched  network/  we  examine  it  as  a case  study  and  attempt 
to  extract  some  general  results  about  designing  a packet 
switched  network  aimed  at  the  ST  class/  but  still  retaining 
capabilities  in  the  other  classes.  The  emphasis  here  is  to 
reduce  delay  in  order  to  satisfy  property  (1).  loop  con- 
trol/ system  priorities/  and  the  effects  of  periodic  routing 
updates  are  among  the  issues  of  concern  under  area  (1). 

1.6  Sjimmat^  gi  Lgsuits 

In  Chafctec  2 we  investigate  the  occurrence  of  loops 
caused  by  routing  update  procedures.  By  routing  we  mean  the 
process  by  which  packets  are  directed  from  switch  to  switch 
through  the  network.  This  definition  is  intended  to  include 
the  structure  and  use  of  local  routing  information  as  well 
as  the  upoating  of  this  local  information.  Routing  algo- 
rithms are  broken  into  four  classes  in  CCrow  75];  (a)  **non- 
adaptive”  or  fixed  - where  the  route  between  any  two  nodes 
remains  fixed.  (b)  "Centralized  adaptive"  - in  which  routes 
are  dynamically  modified  by  a central  overseer.  (c)  "Iso- 
lated adaptive"  - where  routes  are  changed  dynamically  by 
each  node  without  sharing  information  among  the  nodes.  (d) 
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"Distributed  adaptive"  - in  which  routes  are  dynaini  ca  L ly 
changed  by  each  node  with  information  shared  amono  the  nodes 
in  the  form  of  routing  update  packets.  It  should  be  clear 
that  loops  would  be  forbidden  in  types  (a)  and  (b)  by  a 
small  amount  of  care.  It  would  appear  to  be  impossible  to 
prevent  loops  in  type  (c).  We  therefore  concentrate  our 
efforts  in  studying  the  distributed  adaptive  type  algorithm. 
The  existence  of  routing  loops  under  a routing  orocedure  is 
a possible  source  of  performance  degradation.  Packets  which 
are  trapped  in  loops  have  increased  network  delay.  If  a 
loop  persists  sufficiently  long/  then  interaction  is  ham- 
pered if  not  destroyed  (i.e./  property  (1)  is  violated).  In 
CNayl  75]  a loop-free  routing  algorithm  is  presented.  It 
was  shown  that  this  algorithm  cannot  create  loops/  but  its 
operational  characteristics  required  further  examination. 
Ue  investigate  the  operational  characteristics  of  this  and 
other  algorithms  through  the  use  of  simulation.  A local 
loop-free  (or  ping-pong-free)  algorithm  is  found  to  perform 
best  among  those  algorithms  tested. 

In  Chagiat  3 we  consider  priority  assignment  among 
tasks  within  the  system.  An  important  issue  in  the  perfor- 
mance of  a network  is  the  assignment  of  the  priority  among 
the  various  functions  within  a node  as  well  as  the  priori- 
ties assigned  to  the  transmission  of  packets  on  the  chan- 
nels. As  an  example/  we  have  found  that  under  the  current 
ARPANET  strategy/  packets  waited  as  long/  on  the  average/ 
for  the  processor  as  for  the  transmission  channels.  This  is 
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due  to  the  fact  that  routing  update  packets  and  non-routing 
packets  (i.e./  data  and  control  distinct  from  routing  update 
packets)  are  processed  in  f i rst-come-f i rst-served  (FCFS) 
order  within  the  nodes.  In  addition  some  of  the  nodes  in 
the  network  spend  more  than  5QX  of  their  total  capacity  just 
processing  routing  updates^  compared  to  "only"  1F)£  routing 
overhead  on  the  channels.  Examination  of  this  problem  iso- 
lated to  a single  node  suggests  that  since#  in  the  current 
ARPANET/  routing  packets  require  approximately  4C  times  the 
processing  time  that  non- routing  packets  reguire#  the  pro- 
cessing of  routing  packets  should  be  done  at  a lower  priori- 
ty. Simulation  of  the  system  as  a whole#  suggests  that  the 
local  optimum  leads  to  a global  optimum  in  terms  of  mean 
delay  and  network  throughput.  Therefore#  by  modifying  the 
priority  structure  one  could  reduce  delay  and  thereby  assist 
in  complying  with  nroperty  (1). 

In  Chaetfer  4 we  show  some  interesting  delay  behavior 
for  a periodic  stream  traffic  source.  The  behavior  is  due 
to  the  periodic  nature  of  the  routing  update  procedure.  The 
current  ARPANET  scheme  provides  for  periodic  updates#  and 
with  the  growing  size  of  the  network  the  rate  of  routing 
updates  has  been  increased  to  allow  for  better  propagation 
of  information  through  the  network  when  congestion  or 
failure  occurs.  We  show  that  one  must  pay  a high  price#  in 
terms  of  delay#  for  this  feature  during  normal  operation 
however.  Under  normal  conditions  routing  updates  are  not 
required  very  often  since  most  routes  need  not  change  much. 
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Lonq  term  data  on  ARPANET  reliability  shows  that  network 
component  failures  occur  very  infrequently  compared  to  the 
routing  update  period.  These  results  suggest  an  asynchro- 
nous method  of  updating  routing  tables.  This  would  result 
in  lower  average  delay  and  thus  help  provide  property  (1). 

Under  the  second  area  of  emphasis  we  concentrate  on 
methods  which  attempt  to  preserve  the  relative  timing  of  the 
information  without  destroying  property  (1).  In  ChgtiS-E  5. 
we  examine  buffering  strategies/  at  source  and  destination/ 
which  attempt  to  minimize  gaps  in  the  output  stream  and  at 
the  same  time  attempt  to  minimize  the  delay  between  speaker 
and  listener.  Ue  found  that  delay  varies  quite  rapidly  and 
therefore  delay  prediction  could  not  be  used  to  adjust  to 
the  network  dynamics  on  a message  by  message  basis;  rather 
we  found  that  the  sending  strategy  should  remain  fixed  for 
"long"  periods. 

Wore  extensive  results  are  found  to  predict  the  perfor- 
mance of  destination  buffering  schemes.  We  define  some 
delay  prediction  techniques  and  two  playout  methods.  Rased 
on  some  assumptions  on  system  delay  we  have  developed  models 
of  the  system  behavior  in  terms  of  delay  and  gap  probabili- 
ty. The  solution  of  the  models  requires  the  knowledge  of 
the  system  delay  distribution.  Exact  results  are  obtained 
for  the  exponential  distribution  and  shifted  exponential 
distribution.  Numerical  integration  is  used  to  obtain 
results  for  an  Erlang  family  of  distributions  which  previous 
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models  CKlei  64D  tell  us  is  a somewhat  better  model  of  net- 
work delay.  Finally  by  simulation  we  compare  the  perfor- 
mance of  our  delay  variation  estimation  technique  to  a delay 
tracker  method  which  has  been  used  experimentally  in  the 
ARPANET#  and  to  a planned  revision  of  that  scheme. 

£tliEi£E  6 lists  our  conclusions  and  suggests  some  areas 
of  interest  for  further  research. 

Our  critical  examination  of  ARPANET  procedures  has  had 
an  impact  not  only  stream  traffic  communication#  but  on  the 
efficiency  of  packet  switching  in  general.  The  study  of  the 
performance  of  destination  buffering  schemes  has  provided  a 
framework  in  which  other  such  techniques  may  be  examined. 
We  believe  strongly  that  stream  traffic  communication  (re- 
mote voice  communication  in  particular)  is  within  the  realm 
of  uses  for  packet  switched  networks#  but  much  work  is  need- 
ed in  order  to  produce  a usable  system. 


CHAPTER  2 
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LOOP  CONTROL  IN  ADAPTIVE  ROUTING* 


2.1  IOlr2du£iiQn 

In  a packet'switched  network  in  which  some  scheme  of 
adaptive  routing  of  packets  is  used/^  there  exists  the  possi- 
bility that  packets  will  become  trapped  in  loops.  That  is^ 
packets  may  be  routed  in  such  a way  that  they  return  several 
times  to  some  set  of  nodes  for  at  least  a finite  period  of 
time  before  reaching  their  eventual  destination/  thus  wast- 
ing network  bandwidth  and  significantly  increasing  message 
delay.  Routing  loops  are  of  concern  in  stream  communication 
primarily  because  of  this  increased  delay  effect.  In  this 
chapter  we  consider  procedures  for  controlling  such  loops. 
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known 

to 

exist 

for 

some  time 

CKahn 

71]/ 

CFult  7?]/ 

CGerl 

73] 

/ CMcQu 

74]/ 

CCegr  75]/  CPick  76]/ 

CGal  1 

77].  Pre- 

vious  approaches  to  this  problem  have  been  to  detect  and 
remove  loops  CKahn  71D/  CFult  72]/  or  to  reduce  the  in- 
cidence of  such  loops  ChicQu  7AD/  CCegr  75D . Gerla  CGerl  73] 
proved  that  an  optimal  routing  policy  must  be  loop-free. 
More  recently/  with  a procedure  known  as  the  "last  m nodes 
visited"  (LMNV)  algorithm  CPick  76]/  the  packets  are 


• This  chapter  is  a revised  version  of  CNayl  75]. 
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prevented  from  looping  in  loops  of  size  fn  nodes  or  smaller. 

Gallager  has  created  an  optimal  routing  scheme  which  is 
loop-free  C6all  773.  We  first  give  two  examples  of  existing 
networks  whose  adaptive  routing  algorithms  can  cause  such 
loops.  We  next  investigate  the  order  of  magnitude  of  the 
degradation  due  to  loops.  We  then  describe  a new  ping- 
pong-free  and  a new  loop-free  algorithm.  A formalism  for 
dealing  with  the  loop  problem  is  introduced  in  order  to 
prove  the  loop-free  properties  of  the  algorithms.  Then  by 
way  of  simulation  we  compare  the  performance  of  several  of 
the  routing  update  schemes. 

I 

i 

I 

i 

2.2  kxamcis^  Q.t  LsSBIQS 

Below  we  describe  four  routing  update  procedures  at 
least  three  of  which  may  cause  looos  to  occur.  There  is  a j 

degree  of  commonality  among  these  schemes.  In  each  oro-  | 

] 

cedure  there  exists  a routing  table  at  each  node  whose  en-  j 

tries  indicate  the  direction  (i.e.«  the  channel  or  neighbor  | 

address)  in  which  to  send  a packet  headed  for  a particular  j 

1 

destination  (see  Figure  2.1).  When  a packet  arrives  at  a j 

1 

node  (say  node  i)/  one  uses  that  packet's  destination  ad-  ' 

I 

dress  d as  an  index  into  the  routing  table  to  determine  the  ! 

channel  c(i/d)  over  which  to  route  that  packet.  The  differ- 
ences in  these  routing  procedures  apoear  in  the  updating  of 
these  tables. 


5 


2.2.1  &BeAtll£I  B£0£ssju£e£ 

Each  of  the  first  three  procedures  have  been  used  in 
the  ARPANET  CRobe  703#  CMcQu  723.  The  first  two  can  cause 
loops.  The  third  scheme  is  currently  in  operation  and  is 
believed  to  prevent  at  least  ping-pong  loops. 

2. 2. 1.1  taac  P£SQ£  £2UliQ3 

In  the  ARPANET  each  entry  in  the  routing  table  consists 
of  (among  other  things)  a delay  estimate  and  the  address  of 
the  channel  for  which  that  delay  estimate  holds  CMcGu  743r 
CWcGu  74a3/  CKlei  763.  The  table  is  updated  upon  the  ar- 
rival of  a routing  message  from  a node's  neighbor.  The 

routing  message  is  a copy  of  the  delay  portion  of  the 

neighbor's  routing  table.  The  delay  estimates  in  that 
node's  routing  table  are  compared  with  the  routing  table 
delay  estimate  entries  in  the  arriving  update  message  plus 
the  delay  from  that  node  to  its  neighbor  (currently  4 units 
+ 1 unit  for  each  packet  on  that  channel's  queue).  The 

smaller  of  the  two  values  replaces  the  routing  table  esti- 
mate and  the  "best  delay  channel"  is  chanced  if  necessary. 
More  precisely;  Node  i upon  receipt  of  a routing  update 
packet  from  neighbor  node  j performs  the  followino  algorithm 
for  each  d: 

LEE 

1 t'(i»d)  = t(j^d)  ♦ q(i^j)  ♦ h 

2 if  t'(i»d)  < t(i»d)  then  c(i/d)  = c'(i^i) 

3 if  c(i^d)  = c'(i/j)  then  t(i/d)  = t'(i^d) 
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where  t(j/d)  = the  delay  estimate  from  node  j to  node  dr 
q(l^j)  = the  length  (in  packets)  of  the  queue  from  node  i to 
neighbor  node  j/  h = the  one  hop  delay  bias  (currently  4 
units  in  the  ARPANET)^  c(i/d)  = the  (next)  channel  on  which 
to  route  a packet  residing  at  node  i destined  for  node  dr 
and  c*(i^j)  is  the  channel  which  connects  neighbors  i and  j. 
This  is  a simplification  of  the  actual  algorithm^  but  the 
essential  features  are  retained. 

Consider  the  network  topology  pictured  in  Figure  2.2. 
We  are  concerned  with  the  estimated  delay  from  nodes  P and  C 
to  some  distant  node  D.  We  will  use  the  following  notation 
of  CKlei  75a]; 

d/A  d = estimated  delay  to  node  D 
A = next  neighbor  in  path  to  D 

B->C  node  B sends  a routing  update  to  C 


Figurt  2.2.  A portion  of  a network. 


The  following  sequence  may  occur  if  B*s  delay  estimate  to  D 
increases  rapidly  (at  the  second  step  in  the  sequence). 


B->C 


A->B 


C->B 


Node  B 


t+9/A 


t+8/C 


Node  C 


t + 4/B 


P 


B sends  an  update  to  C which  sets  up  the  initial  state. 
Non-empty  queues  are  formed  within  A and  B (B  in  A's  direc- 
tion and  A away  from  B)  when  A sends  an  update  to  B.  This 
increases  0*s  delay  estimate  by  9 units.  C then  instructs  P 
(based  on  B's  old  delay  estimate)  that  the  delay  via  C is 
now  smaller.  This  causes  B to  point  the  route  toward  Cf  but 
C still  points  toward  B.  Therefore  a loop  is  created. 

2. 2. 1.2  il2id”down  rQytiQg  (HDR) 

This  kind  of  loop  was  eliminated  by  the  addition  of  the 
constraint  that  if  the  delay  estimate  changes  by  more  than  ** 
units  on  a given  line  between  updates  then  the  node  enters 
"hold-down"  state  on  that  line  and  will  not  switch  (even 
though  the  delay  estimates  may  be  better  in  another  direc- 
tion) for  approximately  two  seconds.  We  will  not  attempt  to 
explain  this  further  than  to  say  that  it  allows  for  news  of 
drastic  changes  to  propagate  through  a part  of  the  network 
before  routing  changes  are  made.  For  further  details  see 
CKcUu  7A]. 

Let  us  consider  what  may  happen  when  B's  delay  estimate 
to  node  D gradually  increases.  We  assume  that  there  are 
twice  as  many  routing  messages  from  C to  B as  from  B to  C. 
This  is  possible  because  the  rate  at  which  updates  occur  is 
based  on  line  speed  and  line  utilization.  For  a 50  kbps 
line  they  occur  at  a minimum  rate  of  one  every  640  msec,  and 
a maximum  of  one  every  128  msec.  (See  Chapter  4 for  a more 
complete  description.)  We  denote  line  hold-down  by  an 


t 

I 

■ exclamation  point  in  the  followinq  sequence. 


Node  B 

Node  C 

0->C 

t/A 

t+4/B 

A->B 

t+5/A 

CD 

A 

1 

(no  change) 

A->B 

t+9/ A 

C->B 

t+8/  C 

B->C 

t+13/P 

C->B 

t+17/C! 

This  loop  (called  a loop-trap  in  CKlei75a])  will  remain  for 
a period  of  approximately  two  seconds  (the  hold-down  time)! 

^ Conditions  similar  to  this  have  been  observed  on  several 

occasions/  and  account/  at  least  in  part/  for  the  degrada- 
tions suffered  during  experiments  with  stream  commun i cat i on 
I in  the  ARPANET  CCohe  743/  CNayl  743/  CForg  753. 

2. 2. 1.3  iiadififtd  Ii2id“d2«Q  E2ytiQ2 
Notice  that  the  example  of  a loop  under  HDR  is  a result 
of  the  variable  rate  updating.  After  loop-traps  were  ob- 
served CNayl  753/  the  hold-down  scheme  was  modified  in  1975 
to  make  the  criteria  for  entering  hold-down  independent  of 
update  rate.  That  is  to  say  whenever  a delay  estimate 
changes  by  more  than  eight  units  over  an  arbitrary  number  of 
updates  then  hold-down  is  entered.  This  means  that  a gradu- 
al build  up  of  traffic  can  cause  hold-down  to  occur.  In 
fact  hold-down  would  occur  prior  to  many  routing  changes. 


] 

I 


i 
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This  change  may  have  produced  a loop-free  routing  algorithm. 
(U’e  shall  not  attempt  a proof  or  counterexample  here.) 
Surely  some  (possibly  many)  desirable  route  changes  are 
eliminated  or  delayed  by  such  a scheme  however. 

2.2.2  Tjie  lifiAS  NET  Ero££dWIlg. 

This  loop  phenomenon  is  not  unique  to  the  ARPANET/  as 
we  now  show.  The  entries  in  the  routing  table  proposed  for 
the  TIDAS  network  in  CCeqr  75]  contain  a delay  estimate  for 
ea£h  of  the  node's  channels  to  a destination.  The  early 
ARPANET  had  such  tables  as  well  until  the  size  of  the  net- 
work prohibited  them.  When  a packet  arrives/  the  routing 
procedure  chooses  the  channel  with  the  smallest  delay  esti- 
mate (to  the  packet’s  destination)  in  the  table/  and  routes 
the  packet  over  that  channel.  Updates  of  the  table  are  done 
in  two  ways: 

a)  "inside"  (i.e./  using  only  local  information:  queue 

lengths  and  previous  delay  estimates). 

b)  "outside"  (i.e./  using  information  contained  in  a rout- 

ing message  from  one's  neighbor). 

The  routing  message  contains  a delay  estimate  vector  which 
gives  the  delays  from  the  sending  neighbor  but  net  through 
the  receiver  of  the  routing  message.  That  is/  when  A sends 
a routing  message  to  S/  the  second  best  delay  estimate  is 
sent  for  those  destinations  whose  best  delay  channel  is 
toward  B.  This  method  is  referred  to  as  "split  horizon" 
updating  in  CCegr  75].  We  shall  use  a two-component  notation 


similor  to  the  above  with  the  network  structure  shown  in 
Figure  2. id.  The  following  may  occur  with  the  stated  algo- 
rithm. 

Node  Node  C 

t/A/>  t+12/C  t+2/B/  t + 10/E 

A->B  t+13/A,  t+12/C  t+2/B/  t+10/F 

This  loop  may  be  eliminated  with  the  next  routing  message^ 
or  it  may  set  up  a chain  of  loops  in  the  direction  of  E.  We 
are  merely  pointing  out  here  that  loops  form  easily  under 
this  routing  procedure.  In  fact/  in  simulations  of  this 
procedure  "ping  ponging"  was  clearly  present.  According  to 
Cegrell  CCegr  75D  ping-ponging  was  significantly  reduced  by 
the  split  horizon  technique  and  immediate  updating  in  the 
case  of  node  or  line  failures. 

Although  loops  involving  more  than  two  nodes  are  possi- 
ble with  both  of  the  above  schemes/  we  will  not  present  anv 
such  examples.  Fiulti-node  loops  cause  a much  larger  degra- 
dation in  network  performance  than  do  2-node  (ping-pong) 
Loops/  but  their  likelihood  of  occurrence  appears  to  be  sig- 
nificantly smaller  than  that  of  the  two-node  loop.  It  ap- 
pears to  be  impossible  to  prevent  these  loops  when  each  node 
has  only  local  information.  The  feasibility  of  each  node 
carrying  and  adequately  updating  global  information  is  ques- 
t ionable. 

In  the  next  section  we  examine  the  magnitude  of  the 
degradation  in  the  ARPANET  caused  by  the  occurrence  of 


loops 


I ■■Ml 


2.3  DggcaslallSD  ^U£  ti  i.22BS 

Havinq  shown  that  loops  can  occurs  we  now  examine  the 
degradation  to  network  performance  caused  by  loops  in  terms 
of  the  length  of  time  they  naturally  persist.  That  is  to 
say#  we  wish  to  determine  how  Long  it  takes  to  eliminate  a 
loop  under  the  current  ARPANET  routing  update  rates.  This 
persistence  time  is  of  interest  since#  if  loops  disappear 
quickly#  then  there  would  be  no  need  for  prevention  or  rapid 
detection  and  removal.  Me  first  define  the  following  quan- 
tities. 

p = Routing  update  period, 
n = Number  of  nodes  in  the  loop. 

I = Total  local  delay  estimate  in  the  loop  (in  the 
current  implementation  I = A*n#  assuming  the  ab- 
sence of  packets  traversing  the  loop) 
d = minimum  delay  estimate  difference  over  all  neigh- 
bors outside  the  loop, 
k = Processing  time  for  a routing  message. 

X = Transmit  and  propagation  delay  for  a routing  mes- 
sage. 

y = k + X. 

The  Loop  will  be  cleared  whenever  a routing  update# 
from  outside  the  loop#  arrives  and  results  in  a smaller 
delay  estimate  directing  packets  out  of  the  loop.  This  can 
happen  as  quickly  as  it  takes  to  process  an  arriving  routing 
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message^  if  delay  estimates  outside  the  loop  decrease  quick- 
ly. Thus  the  minimum  breaking  time  is  essentially  zero.  If 
we  assume  that  the  delay  estimates  outside  the  loop  remain 
fixed^  then  we  must  traverse  the  loop  with  routing  updates  a 
maximum  of  ceiKd/l)  times  (where  ceil  is  defined  to  be  the 
usual  ceiling  function^  i.e.^  ceil(A)  = the  smallest  integer 
greater  than  or  equal  to  A).  This  causes  the  internal  delay 
estimate  to  be  greater  than  the  external  delay  estimate. 
The  maximum  time  which  a loop  can  persists  assuming  the 
exterior  delay  remains  fixedr  is 

p ♦ y + n<p  + y)ceil(d/l) 

To  compute  a minimum  we  assume  that  the  nodes  are  synchron- 
ized in  such  a way  that  a routing  message  is  sent  just  after 
the  receipt  of  one  from  a neighbor.  We  obtain  for  our 
minimum 

yn(cei  I (d/ 1) ) + (cei  Kd/ l)-1 ) (cei  I (yn/p)p-yn)  + k 

The  first  term  accounts  for  the  correct  number  of  loop 
traversals  required  to  increase  the  internal  delay  estimate 
beyond  the  external  delay  estimate.  The  Irst  term  accounts 
for  the  processing  of  the  one  required  update  from  the 
external  node.  The  middle  term  accounts  for  the  fact  that 
routing  updates  may  happen  no  more  often  than  one  every  p 
msec  over  each  channel.  Table  2.1  shows  the  removal  time 
for  loops  of  size  up  to  ten  nodes^  using  a value  of  25  for  x 
and  15  for  k (see  Chapter  3)  and  assuming  that  the  loop  is 
caused  by  a minimum  delay  estimate  difference  so  that 
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cei I (d/l) 


= 1. 


Loop 

p=640 

msec 

p=128 

msec 

Size 

Max imum 

Minimum 

Maxi  mum 

Minimum 

(nodes) 

(msec ) 

(msec) 

2 

2040 

95 

504 

95 

3 

2720 

135 

672 

135 

4 

3400 

175 

840 

175 

5 

4080 

215 

1008 

215 

6 

4760 

255 

1176 

255 

7 

5440 

295 

1344 

295 

P 

6120 

335 

1512 

335 

9 

6800 

375 

1680 

375 

10 

7480 

415 

18  48 

415 

Table  2.1.  ARPANET  Loop  persistence  time 

In  general  routing  updates  among  neighbors  are  neither 
synchronized  nor  completely  unsynchronized.  Thus  the  ex- 
pected loop  persistence  time  lies  between  the  two  extremes. 
It  is  now  clear  that  loops  do  not  disappear  quickly/  on  the 
average.  Therefore  it  would  be  useful  to  examine  methods  of 
prevention  or  at  least  rapid  detection  and  removal. 


2.4  A di£££l  a|2C£S2£b  logg  £S0ti:&i 

Below  we  describe  two  loop  control  algorithms  for  sin- 
gle entry  routing  tables  (as  in  the  ARPANET).  The  first  is 
ping-pong  or  local  loop-free  and  the  second  is  general 
loop-free . 

2.4.1  A lfi£ai  iSSQ'llSS  ESUllog  aig2ri£hffl  (LL£a> 

The  basic  idea  for  eliminating  local  loops  is  to  ignore 
one's  "old"  information.  Each  node  ignores  routing  informa- 
tion which  points  in  that  node's  direction.  Routing  message 
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entries  are  marked  so  that  the  receiving  node  knows  which 
entries  point  toward  it  and  which  do  not.  Only  those  en- 
tries which  point  away  are  used  in  the  update.  More  pre- 
cisely; we  add  to  each  entry  in  the  routing  update  messag* 
the  best  delay  channel  number  c(j»d)r  and  a neighbor  table  n 
is  appended  to  the  routing  update.  An  entry  n(j/k)  gives 
the  address  of  the  neighbor  connected  to  j on  channel  k. 
(Fntries  could  be  marked  with  a single  bit^  but  this  re- 
quires the  sending  of  a different  routing  message  to  each 
neighbor.)  Each  node  i performs  the  following  algorithm: 


LL£& 

1 m = k such  that  n(j^k)  = i 

2 for  each  d 

3 if  c(j^d)  / m then  do 

4 t' (i^d)  = t( j,d)  + q(3,d)  + h 

5 if  t'(i/d)  < t(i»d)  then  c(i^d)  = c*(i/j) 

6 if  c<i/d)  = c'(i^j)  then  t(i^d)  = t'(i#d) 

7 end 


In  a later  section  we  prove  that  this  algorithm  can 
create  no  local  loops. 


2.4.2  A ifl2B”f£fi£  £2u£lQg  alaQcithoi 
The  loop-free  routing  algorithm  consists  of  LLFR  at 
one-  and  two-connected  nodes  with  the  addition  of  the  fol- 
lowino  for  three-  (or  more)  connected  nodes  (where  a node's 
connectedness  is  defined  to  be  the  number  of  directly 
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connected  neighbor  nodes).  When  a routing  update  indicates 
that  a change  should  occur  at  a three-  (or  more)  connected 
node  a "loop-check"  control  packet  (LCP)  is  sent/  over  the 
new  channel/  one  for  each  destination  which  would  be  effect- 
ed by  this  route  change.  The  idea  being  that  if  the  LCP 
returns  to  the  sender/  then  a loop  would  be  created  by  such 
a routing  change.  If  an  end-to-end  acknowledgement  (PCF) 
for  the  loop-check  packet  returns  to  the  sender/  then  he 
upoates  his  table  and  uses  the  new  route.  Otherwise  (the 
loop-check  packet/  an  " I ' m-check i ng"  packet  (ICP)/  or  noth- 
ing returns)  he  ignores  the  new  route  until  (possibly) 
another  routing  update  is  received.  Loop-check  packets  are 
routed  through  the  network  in  the  usual  way  (i.e./  along 
existing  allowed  paths)  except  that  at  any  node  which  is 
also  checking  a new  route  for  that  same  destination)  such  a 
node  must  send  an  I’m-checking  packet  to  the  source  of  the 
loop-check  packet  and  discard  the  loop-check  packet.  Rout- 
ing message  entries  for  any  destination  site  involved  in 
loop-check  are  specially  marked  so  that  the  information  will 
be  ignored  by  the  receiver  of  the  routing  message.  Stated 
more  precisely  the  assimilation  of  a routing  message  is  done 
under  the  following  algorithm  for  each  d: 


r 


fi 


L£a  CQuiiag  (sfis&aas  i££i!sliail2Q 


1 

if  x(i^d)  = NULL  then  do 

? 

if  c(j^d>  = (!)  then  do 

3 

if  c(i^d)  = c'(i^j)  then  do 

4 

local  loop  detected! 

5 

t (i^d)  = t(  j^d)  + 1 

6 

end 

7 

end 

e 

else  do 

9 

t ’ (i/d)  = t( j /d)  ♦ 1 

10 

if  c'(i^j)  = x(i^d)  then  y(i/d)  = 

t ' ( i /d) 

11 

else  do 

12 

if  t'(ifd)  < y(i*d)  then  do 

13 

y<i#d)  = 0 

U 

x(i/d)  = NULL 

15 

end 

16 

if  t'(i^d)  < t(i#d)  and  x<i/d)  = 

NULL  then 

17 

if  i is  > 2-connected  then  do 

18 

x(i^d)  = c'(i/j) 

19 

y(i^d)  = t'(i^d) 

20 

send  LCP  to  d via  c’(i/j)  with  new  seq 

21 

end 

22 

else  c(i^d)  = c'(i/j) 

23 

end 

24 

end 

25 

if  c(i/d)  = c'(i^j)  then  t(i^d)  = 

t'(i,d) 

26 

end 

where  x(i/d)  is  the  test  channel  for  destination  d at  node 
if  y(i^d)  is  the  test  channel  delay  estimate  to  node  d from 
node  if  m is  defined  as  in  LLFR/  and  I is  the  local  delay 
(e.g.^  I = q(i/})  + h).  Some  additional  processing  of  a 
packet  is  required  as  it  arrives  from  a channel: 

k£B  ssldlllfiQai  &icksl  Q£Q££&SiQa 

101  if  ICP  for  i then  do 

102  if  current  sequence  number  then  do 

103  x(i,d)  = NULL 

10A  y(i/d)  = 0 

105  end 

106  discard  packet 

107  end 

108  if  LCP  then  do 

109  if  source  = i then  do 

110  if  current  sequence  number  then  do 

111  x(i^d)  = NULL 

112  y(i,d)  = 0 

113  end 

114  discard  packet 

115  end 

116  if  destination  = i then  do 

117  send  RCP  to  source 
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discard  packet 
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if  x(i/d)  ^ NULL  then  do 

121 

discard  packet 

122 

send  ICP  to  source 

123 

end 

124 

end 

125 

if  RCP  for  i then  do 

126 

if  current  sequence  number 

then  do 

127 

t(i^d)  = y(i^d) 

128 

c(i^d}  = x(i^d) 

129 

x(i/d)  = NULL 

130 

y(i,d)  = C 

131 

end 

132 

discard  packet 

133 

end 

134 

if  packet  not  discarded  then 

do  normal  packet  processing 

It  is 

useful  to  consider  how  these  algorithms 

wou  Id 

affect  the  ARPANET.  These  algorithms 

appear  to  fit 

easi  ly 

within  the 

current  ARPANET  system  with 

respect  to  the 

fol- 

lowing: 

1.  The  same  routing  message  may  be  sent 

to  each  of  a 

node 's 

neighbors . 

2.  Routing  updates  fit  uithin  the  current  packet  size  res- 
trict i ons . 

3.  The  processing  complexity  is  not  significantly  increased. 
(In  the  case  of  LLFR^  processing  is  certainly  less  com- 
plex than  HDR  or  KHDR.) 
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We  expect  that  the  frequency  of  sending  of  the  I 'm-checki ng 
packet  is  quite  small.  The  frequency  of  both  of  these  new 
types  of  control  messages  may  only  be  tested  by  careful 
measurements  however.  Notice  that  to  avoid  race  conditionss 
one  must  not  send  a routing  update  while  in  the  process  of 
updating  his  own  tables  and  routing  messages  must  not  be 
allowed  to  cross  paths  (i.e.s  sent  from  A to  B and  from  B to 
A in  the  same  time  span).  This  is  easily  dealt  with  by 
using  a pair  of  sequence  numbers  in  the  routing  packets. 
Node  A ignores  B’s  update  unless  it  contains  A's  last  se- 
quence number. 

«e  expect  that  the  performance  of  LLFP  is  better  than 
HORs  since  it  is  less  complex  yet  guarantees  that  no  local 
loops  occur  (as  we  shall  prove).  For  LFR/  the  overhead  is 
much  higher.  There  is  some  extra  computation  reoui red  when 
a packet  arrives.  This  added  computation  is  insignificant 
when  compared  to  the  slowness  of  routing  changes  and  the 
requirement  that  network  bandwidth  be  used  by  the  loop-check 
packets. 

Routing  changes  at  three-  or  more  connected  nodes  can- 
not occur/  on  the  average/  more  quickly  than  about  12  msec 
per  hop  in  the  new  route  (200  bits  in  a loop-check  packet 
and  200  bits  in  an  RCP  gives  8 msec  transmission  delay.  An 
additional  2 msec  is  required  for  processing  of  the  packets. 
Add  to  that  about  2 msec  in  propagation  delay  per  hop.)  The 
average  network  distance  in  May  197A  was  6.24  hops/  with  a 


31 


I 

I 

maximum  of  about  11.  Hence^  one  can  estimate  that  on  the 
averaoe  a minimum  of  75  msec  is  required  to  make  a routing 
change  after  it  has  been  suggested  by  one's  neighbor.  If 
one  assumes  that  loop-check  packets  are  high  priority  (i.e.,^ 
next  to  go  out  on  the  channel)  and  that  a full  packet  has 
^ just  "Started  out  on  the  line  when  the  loop-check  packet 

arrives^  then  the  delay  increases  to  about  60  msec  per  hop. 
This  suggests  that  the  maximum  switch  time  is  on  the  order 
of  700  msec.  Our  experience  shows  that/  on  the  average/  a 
small  number  (<5)  of  destinations  are  involved  when  a switch 
occurs.  At  no  time  would  a three-connected  node  loop-check 
! more  than  about  1/3  of  the  nodes  (less  than  20  nodes  at 

present)  in  the  network  after  a routing  update.  So  that 
between  50  and  220  msec  of  channel  time  and  5 to  20  buffers 
j along  the  way  might  be  utilized. 

An  alternate  method  of  sending  the  loop-check  packets 
could  be  used  to  trade  computation  for  storage  and  line 
utilization.  One  may  send  a mu  1 1 i -dest inat i on  packet  which/ 
along  with  the  usual  packet  header/  contains  some  (A 
currently  in  the  ARPANET)  words  of  addressing  information. 
These  addressing  words  consist  of  one  bit  •^or  each  node  in 
the  network.  The  packet  is  marked  by  the  sender  with  a one 
bit  for  each  destination  for  which  a loop-check  is  required. 
Then  it  is  sent  over  the  new  channel.  Each  node  which  en- 
counters the  packet  is  required  to: 

1.  Send  an  RCP  if  its  own  bit  is  on! 

2.  Turn  its  own  bit  off/ 
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3.  Send  an  ICP  to  the  source  marking  those  destinations 
being  checked  by  both  this  node  and  the  source# 

4.  Turn  off  the  bits  for  those  destinations  being 
checked  by  both  this  node  and  the  source; 

5.  Discard  the  packet  if  all  addressing  bits  are  now 
off; 

6.  (Possibly)  duplicate  the  packet; 

7.  Mark  the  copies  with  those  destinations  pointed  to 
by  each  of  the  channels  as  indicated  in  his  routing 
table;  and 

8.  Forward  the  packet(s)  on  their  way. 

This  method  woi^ld  cut  some  of  the  channel  and  storage  over- 
head at  the  expense  of  some  processing  overhead. 

Pelow  we  present  a proof  that  the  algorithms  do  prevent 
loops.  In  a lat«-r  section  we  present  our  experience  with 
these  algorithms  through  simulation. 

2.5  gl  iQQD-USSDSSS 

There  are  a number  of  definitions  which  facilitate  the 
proving  of  the  loop  prevention  properties  of  these  algo- 
rithms. These  definitions  help  in  the  understanding  of 
these  properties  as  well.  The  algorithm  may  be  described  by 
a graph  structure  in  which  there  are  directed  and  non- 
directed  arcs.  The  non-directed  arcs  are  the  lines  between 
neighboring  nodes.  The  directed  arcs  represent  the  direc- 
tion that  a packet  to  a particular  destination  would  be 
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sent.  There  exist  a set  of  operators  which:  (1)  replace 
directed  arcs  (routing  changes);  (2)  remove  or  create  non- 
directed  arcs  (network  failures  and  recoveries). 


We  define  a network  to  be  a graph  G consisting  of  a 
set  X of  nodes  related  by  three  relations: 

n(eighbor)#  r(oute)^  and  (te)s(t  route) 


If 

x1#  is  a neighbor 

of 

x2 

in 

the  graph 

G then  x1 

i s 

in 

the  set  n(x2)  and 

x2 

i s 

in 

n ( x1 ) . 

A node  x 

is 

k-connected  if  and  only  if  there  are  exactly  k distinct 
nodes  in  n(x).  We  will  refer  to  a k-connected  node  as 
mult i -connected  if  k>2.  The  next  node  in  the  route  to  a 

particular  destination  xri  from  x1  is  r(x1)  . We  shall 
fix  the  destination  xd  here^  and  drop  reference  to  it  for 
simplicity.  All  succeeding  definitions  (and  proofs)  refer 
implicitly  to  a particular  destination.  Loop-check  packets 
from  node  x are  routed  to  the  test  route  s(x)  -^rom  x 
Note  that  s(x)  = NULL  for  all  nodes  except  those  in  loop- 

check  mode.  A loop#  of  order  k # is  defined  to  be  the  fol- 
lowing condition: 

£QDdlli&D  2.1: 

At  some  instant#  there  exists  a set  Y = (y1 #y2# . . . #yk ) # 
k>1#  a subset  of  X#  such  that  xd#  the  destination#  is  not 
in  Y and 


r(yi) 

= y(i+1) 

for  i = 

1#...#  k - 1 and 

r(yk) 

= yi  . 

that 

this  is 

stat i c 

(for  possibly  a short  time) 
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condition  among  the  routing  tables  of  several  nodes.  We  do 
not  intend  to  include  in  this  definition  transient  condi- 


tions under  which  packets  may  actually  loop  (e.g.^  line  or 
node  failures). 

A routing  transformation  T on  G changes  the  route  or 
test  route  of  a single  node  pair,  i.e.^ 

KG)  is  defined  as  follows: 

a)  T(r(x))  = r(x)  for  x not  = xO^ 

T(r(xO))  = r*(xO)  for  some  node  xO^  and 
T(s(x))  = s(x)  for  all  nodes  x;  or 

b)  T(r<x))  = r(x)  for  all  x not  = xQr 

T(r(xO))  = s(xO)  for  some  node  xO^ 

7(s(x))  = s<x)  for  x not  = xO#  and 
T(s(xO))  = null;  or 

c)  T(r(x))  = r(x)  for  all  x not  » xOx 

T(s(x))  = s(x)  for  X not  = xO^  and 

T(s(xO))  = NULL  for  some  node  xO;  or 

d)  T(r(x))  = r(x)  for  all  nodes  x# 

i 

T(s(x))  = s(x)  for  X not  = xO^  and 

T(s(xO))  = s'<xO)  for  some  node  xO.  j 

In  case  (a)  the  route  from  node  xO  is  changed  to  a neighbor 
which  is  not  the  test  route  (which  can  happen  only  at  one- 
or  two-connected  nodes).  In  case  (b)  a new  route  from  node 
xO  is  chosen  to  be  the  test  route  following  a successful 
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test.  In  case  (c)  the  test  route  from  node  xC  is  dropped 
following  an  unsuccessful  test.  In  case  (d)  a test  route 
from  node  xO  is  established.  (Routing  transformations  which 
would  occur  simultaneously  in  an  operating  environment  will 
be  ordered  by  node  number/  say/  and  be  executed  sequentially 
here.)  Me  claim  that  LLFR  and  LFR  have  T with  the  following 
property . 

PCSQ££t3(  2.1: 


8. 

If 

r (x) 

= 

then 

T ( r (y ) ) 

not 

= x; 

b. 

i f 

r (x) 

= V' 

then 

T(s(y)) 

not 

= x; 

c. 

i f 

s (x ) 

= 

then 

T(r(y)) 

not 

= x; 

d. 

if 

s(x) 

= y* 

then 

T(s(v)) 

not 

= X . 

Proof  of  Property  2.1: 

LLFg 

Property  2.1  (a)  is  guaranteed  by  step  3 in  the  LLFR  algo- 
rithm. Property  2.1  <b)/  (c)  and  (d)  are  vacuously  satis- 
fied by  LLFR  since  no  test  routes  are  ever  established. 

LfE  For  nodes  y which  are  two-  (or  less)  connected  no 
test  routes  are  ever  established/  thus  proving  parts  b and 
d.  Part  a is  guaranteed  by  step  8 in  the  LFR  aloorithm. 
Part  c is  similarly  guaranteed  by  step  1.  For  nodes  y which 
are  three-  (or  more)  connected  parts  b and  d are  guaranteed 
by  steps  8 and  1 respectively.  Mult i -connected  nodes  create 
actual  routes  from  successful  test  routes.  huring  the  test- 
ing of  a route  by  a node/  his  neighbor  (along  that  route)  is 
restricted  from  establishing  a new  route  opposite  to  the 
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multi-connected  nodes  as  well. 


Q.E.D. 

LfiSd  H’l'  Loops  of  order  2 cannot  occur  as  a result  of  the 
application  of  LLFR  or  LFR. 

( 

f-  Proof;  This  is  a trivial  consequence  of  Property  2.1.  Sup- 

pose that  there  exists  a loop  of  order  2 after  the  applica- 
tion of  some  T with  Property  2.1.  Then  there  exist  neigh- 
bors X and  y such  that: 

T(r<x)>  = y and  T(r(y))  = x (*) 

In  this  case/  there  are  4 possible  states  prior  to  the  ap- 


plication 

of  T ; 

1) 

r(x)  = y 

and 

r (y ) = 

X/ 

2) 

r(x)  = y 

and 

r(y)  not 

= X/ 

3) 

r(x)  not 

= y 

and  r(y) 

= X/  and 

4) 

r(x)  not 

= y 

and  r(y) 

not  = X. 

State  4 cannot  exist  since  T may  modify  only  one  r (this  is 
where  the  synchronizing  of  neighbors  is  important  in  an 
operating  network).  State  1 is  a previously  existing  loop 
and  hence  was  not  caused  by  T.  States  2 and  3 are  sym- 
metric. Thus  we  consider  only  state  2.  Since  T has  Proper- 
ty 2.1/  and  r(x)  = y / then  T(r(y))  not  = x / in  contrad- 
iction to  the  supposition  at  the  beginning  of  the  proof. 
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Thus  <*)  cannot  occur. 


Q .E.O. 


Ue  have  proved  the  following  theorem. 

Ih£fi£&!I!  The  LLFR  algorithm  is  free  of  local  loops. 

We  now  concentrate  on  LFR. 

Lemma  A one-connected  node  cannot  participate  in  a 

multi-node  loop  (i.e.#  a loop  of  order  k>2). 

Proof:  A one-connected  node  has  only  one  neighbor.  If  a 

one-connected  node  participates  in  a loop  it  must  be  of  ord- 
er 2 . 

0 .E. D. 


LtSISS  2.5:  Loops  of  order  greater  than  2 cannot  be  caused  by 
a routing  change  at  a two-connected  node. 

Proof:  Suppose  a loop  of  order  k > 2 is  created  by  a 
transformation  at  a two-connected  node  y . The  loop  then 
satisfies  Condition  2.1  in  the  following  way: 
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T(r(xi) ) = r(xi)  = y 
T(r(y))  = r*(y)  = x<i  ♦ 1) 

T(r(x(k  - 2))  = r(x(k  - 2))  = x(k  - 1) 

T(r(x<k  - 1))  = r(x<k  - 1)>  = x1 

Also  we  have  that  r(y)  not  = x(i  + 1).  In  fact/’  since  y 
is  two-connected  we  have  r(y)  = xi  . But  r<xi)  = y . This 
is  a pre-existing  loop  of  order  2 which  by  Lemma  2.1  cannot 
occur . 

Q . E. D. 


Lsmms  2.4:  Loops  of  order  k > 2 cannot  be  broken  by  a two 
connected  node. 


Proof:  Suppose  a loop  of  order  k > 2 contains  a node  of 
order  2.  Any  routing  change  at  that  node  alone  would  be  a 
violation  of  Property  2.1. 

O.E. 0. 


39 


Lemma  2.5  shows  that  the  loop-check  packet  has  the  pos- 
sibility of  being  returned  to  its  sender^  if  a loop  would 
have  been  caused  by  that  routing  change. 


Ib£S£&9!  Loops  cannot  occur  as  a result  of  LFR. 

Proof:  We  have  previously  proved  that  2-order  loops  do  not 
occur  (Lemma  2.1)  and  that  k-order  loops  for  k > 2^  are 
independent  of  (i.e.r  cannot  be  caused  or  removed  by)  two- 
connected  (or  one-connected)  nodes  (Lemmas  2.2/  2.3  and 
2.4).  We  shall  now  show  that  loops  of  order  k > 2 cannot  be 
caused  by  transformations  at  multi-connected  nodes.  Suppos* 
that  we  restrict  the  number  of  concurrent  mul t i -connec ted 
node  transformations  to  one.  That  is  to  say/  only  one  node 
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may  test  any  particular  destination.  Suppose  that  a loop 
would  be  created  by  a t ransf ormat i on  at  node  Xf  a multi- 
connected  node.  Since  x is  mult i- connected  it  is  recuired 
to  test  the  new  route  with  a loop-check  packet.  There  are 
four  possible  outcomes  from  the  sending  of  this  loop-check 
packet : 

1.  The  loop-check  packet  returns^ 

2.  An  I'm-checking  packet  returns  (If  some  other  node 
is  checking)/ 

3.  Nothing  returns  (If  the  message  is  tost  or  becomes 
trapped  in  a loop  independent  of  x/  should  one 
occur)/  or 

A.  An  RCP  returns. 

In  cases  1 through  3 no  switch  will  be  made  in  the  route 
from  x/  and  hence  no  loop  erected.  In  case  A we  have  the 
fact  that  the  loop-check  packet  from  x reached  its  destina- 
tion. Since  no  other  routing  changes  occur/  any  of  x's 
packets  will  also  reach  the  destination  by  following  the 
exact  route  that  the  LCP  took.  Therefore  no  loop  exists. 
Hence  a routing  transformation  at  only  one  multi -connected 
node  cannot  cause  a loop. 

we  now  proceed  by  induction  on  the  number  of  concurrent 
mul t i -connected  node  routing  switches.  Assume  that  n-1  or 
less  concurrent  switches  cannot  cause  a loop.  Suppose  n 
concurrent  switches  have  caused  a loop.  Consider  the  ac- 
tivity of  node  x/  that  node  which  is  last  to  start  loop 
check.  Again  there  are  four  possible  responses  to  x's  LCP. 


They  are  as  listed  above.  If  2 or  3 occurs  then  x will 
not  switch  routes  which  contradicts  the  supposition  that  n 
concurrent  switches  caused  the  loop.  Therefore  we  need  only 
consider  response  4.  There  are  two  possible  ways  in  which 
this  may  occur. 

(a)  x's  LCP  arrives  at  none  of  the  other  n-“1  nodes 
inwolved/^  or 

(b>  x's  LCP  arrives  at  (at  least)  one  of  the  other  n-1 
nodes  after  that  node  has  completed  switc'-ing. 

In  case  (a)  the  switch  at  x must  be  independent  of  the  loop. 
No  switch  occurs  at  any  of  the  nodes  which  x's  LCP 

traverses.  Therefore  x cannot  be  part  of  the  loop/  since 

all  packets  from  x will  reach  the  destination  as  did  the 
LCP.  Therefore  the  switch  at  x could  not  have  been  required 
to  form  the  loop.  That  is  the  loop  would  have  been  created 
whether  or  not  x switches.  This  contradicts  the  supposition 

again.  In  case  (b)  we  have  that  the  loop  could  have  been 

caused  by  less  than  n concurrent  switches/  which  contradicts 
the  induction  hypothesis.  Hence  there  exists  no  n such  that 
n concurrent  transformations  at  mu  It i -connected  nodes  causes 
a loop. 

Q. E. D. 


An  adaptive  routing  procedure  should  not  be  vulnerable 
to  network  component  failures  (i.e./  line  or  node  outages). 
We  have  not  shown  that  the  algorithm  is  invulnerable  to  net- 
work failures.  It  is  clear/  however/  that  a loop  cannot  be 
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caused  by  a failure  (i.e.^  the  removal  of  an  arc).  A 
recovery  cannot  cause  a loop  either  because  when  a line 
comes  up  it  has  no  direction  in  a path.  Therefore#  no  loops 
are  created  by  a failure  or  a repair.  However#  on  failure 
(or  repair)  some  routes  must  change.  This  may  cause  flood- 
ing of  the  network  with  LCP's  which  is  clearly  one  of  the 
undesirable  features  of  LFR. 

2.6  SiiguijllQC 

In  order  to  examine  the  performance  of  these  routino 
procedures  we  here  present  the  results  of  simulation.  The 
simulation  was  performed  under  the  following  conditions: 

Topology:  ARPANET  (June  1975)  modified  to  exclude 

satellite  links. 

Lines:  ARPANET  capacities  (mostly  50  kb/s#  some  230.4 

kb/s). 

Nodes:  Infinite  storage#  two  processor  speeds 

(316  - 1.6  usec/cycle  and  516  - .96  usec/cycle). 

Traffic  pattern:  Uniform  traffic  matrix  (i.e.#  same 
intensity  between  all  pairs  of  nodes).  Arrivals 
to  the  network  form  a Poisson  process.  Exactly 
the  same  traffic  was  generated  for  each  of  the 
several  routing  schemes  examined. 

Message  length  distribution:  Exponential  truncated  at 
8C64  bits#  with  a mean  of  12?  bits.  (The  mean  is 
one  half  of  that  reported  in  Cklei  743  to  allow 
for  RFNM's  which  have  zero  text  bits.  Fy  reducing 


the  average  message  size  ve  attempt  to  more  close- 
ly model  the  actual  packet  size  distribution  which 
consists  primarily  of  data  packets  and  RFNM's  in 
the  ARPANET.) 

The  results  of  simulation  are  shown  in  Figure  2.3. 
Network  wide  mean  (one  way)  delay  is  plotted  as  a function 
of  relative  offered  traffic  intensity  (load)  in  this  figure. 
We  observe  that  all  algorithms  produce  the  same  network  wide 
mean  delay  up  to  an  offered  traffic  load  of  5 (5C0  packets 
per  second).  At  10  we  see  that  LPR  blows  up  (i.e.^  average 
delay  > 500  msec).  (Actually  the  simulation  failed  to  reach 
equilibrium  at  this  load.)  This  is  due  to  the  creation  of 
many  loops.  Also  at  load  10  HPR  and  LLFR  perform  better 
than  LFR.  (At  this  load  no  multinode  loops  are  yet  prevent- 
ed). The  most  interesting  result  is  that  while  both  HDR  and 
LFR  blow  up  at  a load  of  14#  LLFP  continues  with  finite 
delay  to  at  least  16  (the  highest  load  simulated).  LLFP 
exceeds  the  performance  both  in  terms  of  delay  and 
throughput  at  all  levels  of  offered  load.  This  is  not  too 
surprising  since  multinode  loops  occur  very  infrequently. 
Therefore  the  overhead  which  LFR  requires  to  guarantee 
loop-f reeness  is  ill  spent  in  the  topology  and  traffic  pat- 
tern of  the  simulation.  Since  local  loops  may  (do)  occur 
with  HDR#  it  performs  worse  than  LLFR  which  prevents  such 
loops . 


Figura  2.3.  Comparativ*  parformanca  of  adaptive  routing  procaduras  (timulation). 
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Tho^e  were  no  Loops  observed  below  a load  of  10  in  this 
simulation.  In  different  topologies  (and  non-uniform  traff- 
ic rates)  it  is  possible  to  create  loops  at  lower  loads.  At 
load  10/  30!(  of  the  routing  changes  at  two-connected  nodes 
resulted  in  local  loops  for  LPR.  HDR  had  no  loops  up  to  a 
load  of  14  at  which  point  21%  of  the  two-connected  node 
routing  changes  resulted  in  local  loops.  The  number  of 
loop-traps  was  unfortunately  not  measured  in  the  simulation. 

As  for  the  frequency  of  packet  looping/  there  was  none 
until  load  10  for  LPR/  load  14  for  HDR/  and  load  15  for 
LLFR.  As  expected/  no  Looping  other  than  LCP's  was  observed 
for  the  LFR  simulation.  The  percent  of  packets  which  looped 
was  4.8/  2.6/  and  .8  respectively.  (The  values  for  LPP  and 
HDR  are  subject  to  question  since  the  simulation  did  not 
reach  equilibrium  in  these  cases.) 

Both  LLFR  and  LFR  prevented  local  loops  begining  at  a 
load  of  10.  LFR  began  to  prevent  multinode  loops  at  a load 
of  12.  The  average  delay  to  receive  an  RCP  increased  with 
load  (as  expected).  The  average  had  a range  of  from  246  to 
275  msec.  The  average  number  of  LCP's  per  test  (i.e./  the 
average  number  of  destinations  involved)  remained  fairly 
small  but  increased  with  load  from  a minimum  of  1.1  to  a 
maximum  of  2.8.  This  suggests  that  mu  1 1 i addressed  LCP's 
would  not  help  efficiency  significantly.  It  also  suggests 
that  LFR  has  the  nasty  habit  of  increasing  its  overhead  when 
the  network  is  heavily  loaded. 

In  the  simulations  we  have  assumed  that  each  of  the 
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routing  update  schemes  requires  the  same  amount  of  process- 
ing as  does  HDR.  This  is  clearly  not  the  case  indicating 
that  at  least  for  LLFR  we  have  shown  a lower  bound  on  per- 
formance with  respect  to  HDR.  It  should  be  noted  that  LLFR 
and  LFR  eliminate  the  need  for  performing  any  computation 
(beyond  the  initial  test)  for  a fraction  1 - 1/k  of  the 
entries  in  the  routing  table  on  the  average^  where  k is  the 
connectivity  of  the  neighbor.  The  issue  of  processing  over- 
head is  discussed  more  fully  in  the  next  chapter. 

2.7  CgQciyjiQQj 

Our  intuition^  which  is  now  supported  by  simulation^ 
tells  us  that  multi-node  loops  occur  very  infrequently  in  a 
"reasonable”  topology  and  traffic  pattern.  Therefore^  since 
LFR  is  so  complicated/  its  practical  worth  is  questionable. 
However/  it  is  interesting  to  know  that  such  a scheme  ex- 
ists. 

Local  loops/  on  the  other  hand/  occur  much  more  fre- 
quently and  in  fact  were  observed  in  the  ARPANET  as  well  as 
in  simulation.  Therefore  LLFR  is  of  practical  significance. 
Indeed  the  simulation  results  indicate  that  LLFR  is  the  best 
among  the  procedures  studied  so  far.  In  Chapters  3 and  A 
other  routing  procedures  are  investigated.  We  defer  our 
recommendations  regarding  routing  until  after  Chapter  A. 
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CHAPTER  3 
SYSTEM  PRIORITIES 


3-1  10l£fidU£tifiD 

The  Inclusion  of  levels  of  priority  within  a system 
allows  that  system  to  give  different  levels  of  service  to 
different  classes  of  customers.  This  is  a useful  property 
of  a system  which  is  to  support  a variety  of  activities 
simultaneously.  The  ARPANET/  for  example/  was  originally 
designed  to  support  both  interactive  (terminal-to-computer 
and  computer-to-terminal ) traffic  as  well  as  moving  large 
amounts  of  data  from  place  to  place  in  the  network 
( computer-to-computer  communication).  BBN  has  thoughtfully 
provided  a priority  structure  within  the  network  nodes.  In 
this  chapter  we  partially  examine  that  priority  structure 
and  its  effect  on  performance.  The  result  of  this  examina- 
tion points  out  the  general  rule  that  careful  consideration 
must  be  given  to  priority  at  aii  points  of  service  within 
the  system  in  order  to  guarantee  the  desired  Level  of  ser- 
vice to  each  customer  class. 

Initial  analysis  and  design  of  packet  switched  networks 
assumed  that  the  nodal  processing  time  was  small  and  fixed 
CKlei  64D/  CFran  70]/  CHear  703/  Cklei  70]/  CFran  723  and 
others.  Often/  in  fact/  this  contribution  to  delay  has  been 
neglected  in  comparison  to  the  time  spent  waiting  for  and 


using  the  channels  in  the  network.  In  shorts  the  transmis- 
sion lines  were  the  bottleneck  in  the  system.  The  ARPANET 
in  fact  was  designed  with  this  underlying  assumption.  (One 
must  in  general  attempt  to  fully  utilize  the  most  expensive 
component  in  a system.)  Here  we  present  measurements  which 
demonstrate  that  nodal  processing  time  is  no  longer  small  or 
fixed  in  the  current  ARPANET.  Nodal  processing  delay  is  in 
fact  a significant  portion  of  the  overall  network  delay. 
This  need  not  be  the  case.  The  problem  is  one  of  assigning 
the  proper  priority  to  tasks  within  the  processor.  This  we 
show  through  the  use  of  analysis  and  simulation. 

3.2  ci£a£ui:£d  asiuili 

A large  increase  in  network  wide  mean  round-trip  delay 
was  noticed  between  the  "weeklong"  data  collections  of  Au- 
gust 1973  reported  in  CKlei  74]  and  May  1974.  While  the 
traffic  characteristics  did  not  change  si gni f i cant ly^  delay 
increased  from  93  to  249  msec.  This  delay  (while  fairly 
small)  exceeds  the  ARPANET  specification  of  200  msec  and  has 
remained  at  about  this  level  even  as  late  as  March  1977  when 
another  collection  revealed  a mean  delay  of  228  msec.  Table 
3.1  summarizes  the  information  derived  from  the  three  week- 
long  collections.  There  are  three  factors  (listed  in  the 
table)  which  would  naturally  have  caused  an  increase  in 
delay  between  August  1973  and  May  1974.  Notice  that  in  May 
1974  (a)  messages  contained  slightly  more  packets  on  the 
average^  (b)  messages  traveled  longer  distances  (in  hops)  on 


the  average  and  (c)  the  mean  channel  utilization  (with  over- 
head) was  larger  than  in  August  1973.  The  reason  for  the 
sharp  increase  in  channel  utilization  is  due  to  the  fact 
that  the  frequency  of  routing  updates  was  increased  approxi- 
mately by  a factor  of  five.  All  of  these  three  factors  are 
accounted  for  in  the  model  of  CKtei  743.  Vet  the  model  is 
unable  to  predict  all  of  the  increase.  While  actual  delay 
increased  by  168%  and  145%/  the  model’s  predictions  in- 
creased by  only  95%  and  44%  from  August  1973  to  May  1974  and 
from  August  1973  to  March  1977  respectively.  This  is  due 
(at  least  in  part)  to  the  fact  that  the  model  of  CKlei  743 
assumed  a small  and  fixed  nodal  processing  delay  which  is  no 
longer  true. 


August  1973 

May  1974 

Input  rate  (pkts/sec) 

51 

44 

Mean  bits/msg 

243 

234 

Mean  pkts/msg 

1.11 

1.12 

Mean  traffic  weighted 
shortest  hop  path 

3.24 

4.46 

Mean  channel 
uti 1 ization 
with  overhead 

.071 

.204 

March  1977 
10^ 

266 
1 .18 

2.93 


.237 


Mean  channel 
util ization 

without  overhead  .0077  .011  .012 

Mean  measured 

round-trip  delay  (msec)  93  249  228 

CKlei  743  model 

delay  prediction  (msec)  73  142  101 

Table  3.1  ARPANET  traffic  and  delay  summary 
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In  examining  the  results  of  measurement  experiments 
involving  the  use  of  packet  trace  Cklei  76a]/  it  becomes 
clear  that  the  processing  delay  has  a measured  value  which 
is  much  larger  than  the  available  estimates.  Some  examples 
follow.  Table  3.2  shows  the  results  of  an  experiment  in 
which  packet  trace  was  collected  with  a freguency  of  256 
(i.e./  a trace  block  was  generated  for  every  Z56th  racket 
traversing  a node)  from  several  nodes  in  the  ARPANET.  The 
data  was  collected  during  two  approximately  20  hour  inter- 
vals in  March  1975.  Statistics  for  individual  output  lines 
are  included  (though  one  would  not  expect  processing  delay 
to  vary  with  output  line).  The  first  six  column  headings 
are  clear/  but  the  last  three  require  some  explanation. 
Smpl-Size  (sample  size)  refers  to  the  actual  number  of  pack- 
ets used  to  arrive  at  the  previously  listed  statistics 
(i.e./  Mean/  SDev/  Min/  Max).  The  mean/  standard  deviation/ 
minimum/  and  maximum  are  expressed  in  milliseconds  (msec). 
Only  sto re-and-f orwa rd  packets  which  did  not  originate  at 
the  From-Node  may  be  included  in  the  Smpl-Size/  as  packets 
from  a HOST  do  not  have  their  "time-in"  recorded  properly 
and  thus  are  excluded  from  the  statistics.  They  are/  howev- 
er/ included  in  the  To-Ch-Tot  (to  channel  total)  which  gives 
the  total  number  of  observed  packets  to  be  routed  on  that 
output  channel.  The  To-Task-Tot  (to  Task  total)  entry  is 
the  total  number  of  observed  packets  ( store-and- fo rwa r d and 
reassembly)  which  were  processed  by  the  "Task"  CMcQu  723 


routine  in  the  IMP 


From 

Node 

To 

Node 

Mean 

SOev 

Mi  n 

Na  X 

Smpl 

Size 

To-Ch 

T ot 

T 0-Task 
Tot 

6 

WIT 

31 

CCAT 

2.97 

4.20 

.4 

36.1 

3349 

3724 

8958 

6 

MIT 

47 

WPAT 

3.12 

4.46 

.4 

36.4 

3260 

4001 

8958 

6 

MIT 

44 

MIT2 

3.36 

4.68 

.4 

28.3 

230 

402 

8958 

5 

BBN 

31 

CCAT 

3.10 

4.51 

.4 

33.9 

2852 

3119 

8065 

5 

BBN 

50 

RCC 

3.06 

4.21 

.4 

25.0 

1345 

1746 

8065 

5 

BBN 

49 

RCCT 

3.15 

4.39 

.4 

30.4 

1581 

2149 

8065 

U 

CMU 

38 

PURD 

3.61 

4.73 

.4 

32.6 

1959 

2329 

5930 

14 

CMU 

27 

BELV 

3.13 

4.60 

.4 

33.3 

2071 

2165 

5930 

14 

CMU 

18 

RADT 

3.31 

4.74 

.4 

23.8 

645 

674 

5930 

29 

A6RD 

46 

RUTT 

2.67 

3.78 

.4 

30.6 

1592 

1610 

4720 

29 

ABRD 

19 

NBST 

2.67 

3.67 

.4 

23.3 

1691 

1693 

4720 

29 

ABRD 

27 

BELV 

3.02 

4.00 

.4 

23.9 

1346 

1396 

4720 

11 

STAN 

16 

AMES 

2.14 

3.15 

.4 

26.0 

2027 

2484 

5020 

11 

STAN 

22 

ISI 

2.07 

2.96 

.4 

23.4 

1935 

2180 

5020 

22 

ISI 

11 

STAN 

3.43 

5.39 

.4 

38.1 

572 

1437 

9096 

22 

ISI 

48 

AFWT 

3.59 

4.60 

.4 

27.3 

544 

1311 

0096 

22 

ISI 

52 

ISIT 

3.46 

4.83 

.4 

26.6 

577 

3683 

9096 

1 

UCLA 

8 

SOC 

2.81 

4.13 

.3 

29.5 

1210 

1486 

4346 

1 

UCLA 

3 

UCSB 

2.90 

3.78 

.4 

20.7 

937 

1001 

4346 

1 

UCLA 

35 

UCSO 

2.73 

3.72 

.3 

22.6 

1121 

1508 

4346 

16 

AMES 

15 

AMST 

4.19 

5.81 

.3 

40.6 

983 

1282 

4983 

16 

AMES 

45 

MOFF 

3.71 

4.84 

.4 

26.8 

12«4 

1403 

4983 

16 

AMES 

36 

HAWT 

3.76 

5.16 

.4 

31  .6 

410 

463 

4983 

16 

AMES 

11 

STAN 

4.18 

5.40 

.4 

31.5 

1427 

1543 

4983 

2 

SRI 

51 

SRI3 

3.02 

4.08 

.4 

25.5 

578 

1069 

4852 

2 

SRI 

32 

XROX 

2.96 

4.15 

.4 

27.6 

1113 

1301 

4852 

2 

SRI 

21 

LLL 

2.89 

4.20 

.4 

24.8 

1041 

1728 

4852 

10 

LL 

44 

MIT2 

1.80 

2.77 

.4 

14.3 

73 

119 

265 

10 

LL 

18 

RADT 

2.46 

4.00 

.4 

26.4 

81 

94 

265 

Table  3.2(a).  Observed  processing  delay  in  msec 

(516  I»1Ps) 
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0HI 


From 

Node 

To 

Node 

Mean 

SDev 

Min 

Max 

Smp  1 
Size 

To-Ch 

Tot 

To-Ta  sk 
Tot 

34 

lbl 

21 

LLL 

11.5 

11.3 

.7 

62.2 

3485 

34  92 

10147 

34 

LBL 

4 

UTAT 

10.6 

10.8 

.7 

67.7 

4867 

4956 

1C147 

34 

LBL 

45 

MOFF 

12.2 

11.5 

.7 

64.0 

1477 

1610 

10147 

4 

UTAT 

34 

LBL 

17.1 

30.7 

.7 

323.0 

4769 

5017 

10534 

4 

UTAT 

12 

ILL 

17.1 

31.2 

.7 

333.2 

4693 

5102 

10534 

43 

TYMT 

32 

XROX 

18.2 

16.8 

.7 

90.9 

274 

2724 

5200 

43 

TYMT 

33 

FNWT 

18.2 

17.1 

.7 

71.2 

222 

630 

5200 

23 

USCT 

8 

SDC 

12.2 

13.1 

.7 

76.2 

800 

1647 

4559 

23 

USCT 

25 

DOCT 

12.6 

13.0 

.7 

81.6 

990 

1559 

4559 

25 

OOCT 

23 

USCT 

10.8 

11.7 

.7 

62.9 

1041 

1054 

2096 

25 

DOCT 

24 

GWCT 

11  .4 

11.7 

.7 

62.8 

1020 

1038 

2096 

28 

ARPT 

20 

ETAT 

9.70 

11.1 

.6 

54.7 

351 

644 

1114 

28 

ARPT 

17 

MTRT 

10.6 

10.6 

.7 

50.6 

287 

308 

1114 

27 

BELV 

14 

CMU 

9.88 

10.0 

.7 

57.8 

1247 

1300 

2791 

27 

BELV 

26 

SOAC 

9.94 

10.8 

.7 

52.3 

391 

391 

2791 

27 

BELV 

29 

ABRD 

9.18 

9.94 

.7 

51.9 

1061 

1073 

2791 

44 

MIT2 

6 

MIT 

5.47 

6.64 

.7 

32.5 

113 

219 

391 

44 

MIT2 

10 

LL 

4.67 

5.87 

.7 

26.5 

125 

127 

391 

Table  3.2(b}.  Observed  processing  delay  in  msec 

(316  IMPS  8 TIPs) 


Only  a portion  of  the  total  processing  delay  is  meas- 
ured by  the  trace  mechanism#  viz:  (a)  part  of  the  Modem-to- 
IMP  routine  processing  time#  (b)  the  time  spent  waiting  on 
queue  for  the  Task  routine#  and  (c)  most  of  the  Task  routine 
processing  time.  The  sum  of  (a)#  <b)#  an  (c)  is  given  by 
the  difference  between  T<2)  and  T(1)  in  the  trace  block.  Bv 
the  estimates  in  CMcQu  723  this  value  would  have  a minimum 
of  less  than  150+250=400  cycles  (approximately  .3P  msec  for 
a 516  IMP  and  .64  msec  for  a 316  IMP).  The  observed  minimum 
values  confirm  these  estimates. 
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Table  3.2  has  been  separated  Into  two  parts  with  516 
IMPS  Listed  first  followed  by  the  316  IMPs.  There  is  a 
clear  distinction  between  these  processor  types  (not 
surprising  since  one  runs  at  about  1.67  times  the  speed  of 
the  other).  (Not  as  clear  a difference  was  observed  between 
those  nodes  with  VDH  CBBN  69D  and/or  TIP  COrns  723  software 
and  those  without.)  In  both  cases  we  see  a wild  variation  in 
processing  delays  with  mean  values  from  4.5  to  10.5  times 
the  minimum  for  516s  and  from  6.7  to  26  times  the  minimum 
for  316s.  This  phenomenon  is  nearly  independent  of  time  of 
day  as  shown  in  Tables  3.3  and  3.4^  which  show  the  observa- 
tions of  two  selected  nodes  over  seven  time  intervals.  (The 
rather  odd  time  intervals  correspond  to  arbitrary  file  boun- 
daries and  provide  similar  sample  sizes.)  Notice  again  the 
large  variation  in  processing  delay  and  the  very  high  mean 
values.  Notice  also  that  the  mean  values  remain  high  even 
under  the  more  lightly  loaded  evening  hours#  indicating  (but 
not  proving)  that  this  effect  is  not  due  to  the  competition 
of  packets  for  the  services  of  the  Task  routine. 

For  the  model's  predictions  in  Table  3.1  we  have  as- 
sumed an  average  processing  time  of  less  than  1 msec  per 
hop.  Taking  a rough  average  of  the  values  in  Table  3.2  we 
find  that  average  processing  delay  for  516  IMPs  is  about  3 
msec  and  about  11  msec  for  316  IMPs.  Roughly  one  third  of 
the  IMPS  in  May  1974  were  516  IMPs.  Assuming  a uniform 
spread  of  the  traffic#  the  average  nodal  processing  delay  is 
more  than  8 msec.  If  we  add  7 msec  per  hop  to  the  predicted 
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delays/  Me  would  expect  to  obtain  a better  prediction. 
Doing  so  we  arrive  at  a value  of  204  msec  for  May  1974  and 
142  for  March  1977.  For  May  1974  this  new  prediction  is 
slightly  better  than  was  the  original  prediction  for  August 
1973.  The  March  1977  prediction  is  not  as  good.  One  reason 
may  be  the  flow  control  mechanism  which  was  changed  signifi- 
cantly between  May  1974  and  March  1977.  Delay  is  measured 
in  such  a way  to  include  the  delay  due  to  flow  control/ 
while  there  is  no  provision  for  this  in  the  model. 

From  To  Mean  SDev  Min  Max  Smpl  To-Ch  To-Task  Time 


Node 

Node 

Size 

Tot 

Tot 

Begin 

End 

22 

ISI 

11 

STAN 

4.67 

7.12 

.4 

38.1 

118 

254 

1404 

0952 

1111 

22 

ISI 

4S 

AFWT 

4.06 

4.58 

.4 

17.5 

82 

286 

1404 

0952 

1111 

22 

ISI 

52 

ISIT 

3.00 

3.76 

.4 

15.9 

125 

484 

1404 

0952 

1111 

22 

ISI 

11 

STAN 

4.24 

5.87 

.4 

27.3 

56 

219 

1447 

1111 

1246 

22 

ISI 

48 

AFWT 

4.13 

4.78 

.4 

17.9 

72 

334 

1447 

1111 

1246 

22 

ISI 

52 

ISIT 

3.66 

5.83 

.4 

26.6 

52 

432 

1447 

1111 

12  46 

22 

ISI 

11 

STAN 

2.50 

3.53 

.4 

13.5 

55 

205 

1245 

1246 

1421 

22 

ISI 

48 

AFWT 

3.31 

4.51 

.4 

21.3 

82 

221 

1245 

1246 

1421 

22 

ISI 

52 

ISIT 

3.99 

4.93 

.4 

26.1 

75 

479 

1245 

1246 

1421 

22 

ISI 

11 

STAN 

3.58 

5 .40 

.4 

26.2 

50 

223 

1273 

1421 

1628 

22 

ISI 

48 

AFWT 

4.71 

5.94 

.4 

25.3 

34 

86 

1273 

1421 

1628 

22 

ISI 

52 

ISIT 

3.55 

5.36 

.4 

26.3 

76 

581 

1273 

1421 

1628 

22 

ISI 

11 

STAN 

3.27 

5.43 

.4 

34.0 

86 

219 

1387 

1628 

1853 

22 

ISI 

48 

AFWT 

3.50 

4.85 

.4 

27.3 

97 

141 

1387 

1628 

1853 

22 

ISI 

52 

ISIT 

2.28 

3.28 

.4 

17.0 

54 

585 

1387 

1628 

1853 

22 

ISI 

11 

STAN 

2.37 

4.20 

.4 

30.3 

105 

174 

1517 

1853 

2325 

22 

ISI 

48 

AFWT 

2.99 

3.92 

.4 

17.8 

104 

149 

1517 

1853 

2325 

22 

ISI 

52 

ISIT 

4.42 

6.06 

.4 

26.4 

95 

743 

1517 

1853 

2325 

22 

ISI 

11 

STAN 

3.08 

4.15 

.4 

20.5 

91 

143 

823 

2325 

0529 

22 

ISI 

48 

AFWT 

3.29 

4.38 

.4 

25.7 

73 

94 

823 

2325 

0529 

22 

ISI 

52 

ISIT 

3.15 

4.18 

.4 

20.0 

97 

379 

823 

2325 

0529 

Table  3.3.  Variation  of  processing  delay  with  time  of  day 

<516  IMP) 
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From  To  Mean  SOev  Min  Max  Smpl  To-Ch  To-Task  Time 


Node 

Node 

Size 

Tot 

Tot 

Begin  End 

27 

BELV 

14 

CMU 

7.07 

7.27 

.7 

29.8 

140 

153 

341 

0952 

1111 

27 

BELV 

26 

SDAC 

7.71 

7.37 

.7 

25.8 

81 

81 

341 

0952 

1111 

27 

BELV 

29 

ABRD 

8.66 

8.72 

.7 

40.7 

98 

101 

341 

0952 

1111 

27 

BELV 

14 

CMU 

10.9 

11.0 

.7 

49.7 

165 

176 

431 

1111 

1246 

27 

BELV 

26 

SDAC 

11.5 

11.5 

.7 

38.8 

114 

114 

431 

1111 

1246 

27 

BELV 

29 

A6RD 

10.3 

10.8 

.7 

42.7 

130 

131 

431 

1111 

12  46 

27 

BELV 

1 4 

CMU 

10.2 

11.5 

.7 

57.8 

214 

231 

485 

1246 

1421 

27 

BELV 

26 

SDAC 

12.7 

13.1 

.7 

52.8 

92 

92 

485 

1246 

1421 

27 

BELV 

29 

ABRD 

12.6 

13.2 

.7 

51.9 

153 

155 

485 

1246 

1421 

27 

BELV 

14 

CMU 

9.24 

9.61 

.7 

45.1 

128 

132 

256 

1421 

1628 

27 

BELV 

26 

SDAC 

6.94 

7.82 

.7 

22.1 

20 

20 

256 

1421 

1628 

27 

BELV 

29 

ABRD 

7.28 

7.90 

.7 

29.9 

102 

104 

256 

1421 

1628 

27 

BELV 

14 

CMU 

10.0 

9.69 

.7 

51.7 

194 

402 

1628 

1853 

27 

BELV 

26 

SDAC 

5 .09 

5.53 

.7 

16.0 

17 

17 

402 

1628 

1853 

27 

BELV 

29 

ABRD 

7.84 

7.93 

.7 

34.7 

1 87 

187 

402 

1628 

1853 

27 

BELV 

14 

CMU 

9.61 

9.11 

.7 

40.6 

166 

168 

356 

1853 

2325 

27 

BELV 

26 

SDAC 

12  .6 

13.2 

.7 

49.4 

27 

27 

356 

1853 

2325 

27 

BELV 

29 

ABRD 

8.38 

8.84 

.7 

43.0 

159 

161 

356 

1853 

2325 

27 

BELV 

14 

CMU 

10.9 

9.97 

.7 

41.0 

246 

246 

520 

2325 

0529 

27 

BELV 

26 

SDAC 

5.60 

6.30 

.7 

21  .0 

40 

40 

520 

2325 

0529 

27 

BELV 

29 

ABRD 

8.73 

9.83 

.7 

45.7 

232 

234 

520 

2325 

0529 

Table  3.4.  Variation  of  processing  delay  with  time  of  day 

(316  IMP) 


In  the  previously  described  exoeriment  it  was  not  pos- 
sible to  empirically  prove  that  the  excessive  delay  is  not 
due  to  the  interference  of  packets^  since  sampling  was  done. 
To  establish  this/  we  resort  instead  to  another  experiment 
in  which  we  selected  two  nodes  and  collected  packet  trace 
with  a frequency  of  one.  On  close  examination  of  the  data 
we  find  that  for  most  packets  having  a long  processing  delay 
no  other  packet  overlaps  it  in  time,  we  note  that  while 


some  overflows  occurred  (i.e./  lost  data  due  to  heavy  traff- 


ic) in  the  packet  trace  data  the  statistical  properties  are 
nearly  identical  to  those  of  the  sampled  data.  We  conclude 
that  most  of  the  time  the  excessive  delay  is  not  due  to  the 
interference  of  other  packets. 

At  first  glance/  these  long  processing  delays  seem 
rather  insignificant  in  the  entire  scheme  of  things.  Let's 
examine  them  a bit  more  closely.  Table  3.5  gives  the  aver- 
ages for  processing  delay/  waiting  time/  and  acknowledgement 
time/  for  three  packet  types  - (1)  subnet  control  (largely 
end-to-end  acknowledgements/  i.e./  RFNMs)/  (2)  user  priori- 
ty/ and  (3)  non-priority  packets.  Our  interest  is  with  the 
store-and-f orward  delays/  but  we  list  the  reassembly  infor- 
mation for  completeness.  The  sample  sizes  listed  consist  of 
two  numbers.  The  first  number  indicates  the  sample  size  for 
the  Channel  Waiting  and  Acknowledgement  statistics/  and  the 
second  refers  to  the  Processing  statistics.  Notice  that 
the  time  spent  waiting  for  and  being  served  by  the  Task 
routine  is  on  the  same  order  as  the  time  spent  waiting  for 
the  channel  for  the  516  IMP  and  always  greater  (about  dou- 
ble) in  the  case  of  the  316  IMP.  We  note  that  for  these 
measurements/  non-priority  packets  traversed  a 316  node  fas- 
ter/ on  the  average/  than  both  control  packets  and  priority 
packets  (the  order  of  service  by  Task  is  FCFS)  even  though 
the  channel  waiting  time  is  greater  (as  one  would  expect) 
for  non-priority  packets!  Examination  of  more  extensive 
data  reveals  that  10  of  36  cases  exhibit  this  behavior.  In 


fact  in  only  16  of  38  cases/  including  all  node  types/  was 
processing  delay  smaller  for  both  priority  and  control  pack- 
ets. The  experiments  which  we  have  conducted  in  transmit- 
ting simulated  speech  data  from  the  UCLA  PDP  11/45  [Nayl  741 
as  well  as  similar  experiments  conducted  by  the  University 
of  Southern  California  Information  Sciences  Institute/  Mari- 
na Del  Key/  California  (ISI)  CCohe  74D  support  this  observa- 
tion. In  those  experiments  no  discernible  difference  in 
delay  could  be  found  between  priority  and  non-priority  mes- 
sages! In  a report  from  Network  Analysis  Corporation/  Glen 
Cove/  New  York  (NAC)  CNAC  751  the  authors  state  that  a 40*^ 
decrease  in  mean  round-trip  delay  can  be  achieved  by  reduc- 
ing the  processing  delay  to  its  original  estimate  of  1 msec. 
Our  own  projections  (which  appear  in  Section  3.3.5)  show 
that  a 44X  decrease  could  be  achieved.  Therefore  processing 
delay  currently  accounts  for  at  least  2/5  of  the  total  de- 
lay/ which  is  quite  significant. 


Store-and- Forward 

Reassembly 

Mean  SOev 

Mean 

SDe  V 

Control  Packets 

Sample  Size 

( 3790*  1116) 

( 0, 

0) 

Process ing 

3.81  5.22 

0.0 

0.0 

Channel  Waiting 

2.31  10.40 

0.0 

n.o 

Acknowl edgement 

19.11  15.56 

0.0 

0.0 

User  Priority  Packets 

Sample  Size 

( 731,  317) 

( 2098, 

2066) 

Process ing 

2.57  3.83 

3.36 

4.69 

Channel  Waiting 

2.69  14.41 

5.87 

109.68 

Acknowledgement 

22.65  15.03 

4.36 

18.75 

User  Non-priority 

Packets 

Sample  Size 

( 1910,  260) 

( 567, 

468) 

Pro  cess ing 

3.18  4.95 

8.27 

27.69 

Channel  Waiting 

3.88  26.64 

12.20 

34.87 

Acknowledgement 

22.93  16.87 

6.12 

11.04 

Table  3.5(a).  Processing  delay  statistics 

(node  22  a 516  IMP^  U 

MAR  75) 

Stor e-and- 

■Forward 

Reassembly 

Mean 

SDev 

Mean 

SDev 

Control  Packets 

Sample  Size 

( 1546, 

1507) 

( 0, 

0) 

Processing 

10.43 

10.30 

0.0 

0.0 

Channel  Waiting 

4.89 

8.80 

0.0 

0.0 

Acknowl edgement 

20.73 

11.16 

0.0 

0.0 

User  Priority  Packets 

Sample  Size 

( 575, 

575) 

( 11, 

11 ) 

Processing 

9.63 

10.36 

14.82 

10.78 

Channel  Waiting 

5.00 

8.60 

0.36 

0.50 

A cknow 1 edgement 

23.12 

10.94 

3.64 

5.43 

User  Non-priority 

Packet  s 

Sample  Size 

( 643, 

617) 

( 16, 

16) 

Process ing 

7.63 

9.08 

16.38 

22.62 

Channel  waiting 

5.86 

12.68 

1.56 

3.35 

Acknowl edgement 

26.22 

12.61 

5.13 

5.06 

Table  3.5(b).  Processing  delay  statistics 

(node  27  a 316  IMP^  U MAR  75) 
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« 


I 

1 


Store-and-Forward 

Reassembly 

Mean  SOev 

Mean 

SOev 

Control  Packets 

Sample  Size 

( mi,  2738) 

( 0/ 

0) 

Process ing 

3.75  4.78 

0.0 

0.0 

Channel  Uaiting 

2.72  15.23 

0.0 

P.O 

Acknowl edgement 

21.43  13.04 

0.0 

P.O 

User  Priority  Packets 

Sample  Size 

( 1328/  423) 

( 4344/ 

4326) 

Process ing 

2.92  4.12 

4.14 

4.98 

Channel  Waiting 

3.07  22.03 

0.34 

0.61 

Acknowledgement 

24.16  14.37 

2 .50 

0.97 

User  Non-priority 

Packet  s 

Sample  Size 

( 4834/  2356) 

( 613/ 

611  ) 

Process ino 

3.42  4.64 

4.81 

5 .44 

Channel  Waiting 

5.01  17.04 

0.44 

0.97 

Acknowl edgement 

27.20  13.66 

4.58 

3.37 

Table  3.5(c).  Processing  delay  statistics 

(node  22  a 516  IMP^  24 

JUL  75) 

Store-and 

-Forward 

Reassemt  ly 

Mean 

SOev 

Mean 

SOev 

Control  Packets 

Sample  Size 

( 4317/ 

4258) 

( 0/ 

0) 

Processing 

8.12 

8.39 

0 .0 

0.0 

Channel  Waiting 

3.33 

6.63 

0.0 

0.0 

Acknowledgement 

19.74 

11.33 

0.0 

c.o 

User  Priority  Packets 

Sample  Size 

( 1 543/ 

1543) 

( 4/ 

4) 

Processing 

8.47 

8.16 

3.75 

4.86 

Channel  Waiting 

3.56 

7.79 

0.75 

0.50 

Acknowledgement 

22.24 

10.52 

2.50 

1 .73 

User  Non-priority 

Packets 

Sample  Size 

( 1737/ 

1700) 

( 3/ 

3) 

Process ing 

6.58 

7.85 

8.33 

7.02 

Channel  waiting 

4.77 

8." 

1 .00 

0.0 

Acknowl edgement 

23.42 

10.22 

1 .00 

0.0 

Table  3.5(d).  Processing  delay  statistics 

(node  27  a 316  IMP^  24  JUL  75) 


The  acknowledgement  time  is  larger  than  one  might  ex- 
pect. This  can  have  only  a second  order  effect  on  delay. 
If  packets  are  retained  longer  than  necessary  when  buffers 
are  in  short  supply  then  some  packets  may  have  to  be 


retransmitted.  This  is  not  occurring^  for  if  it  were  the 
effect  would  be  recorded  as  waiting  time  since  retransmis- 
sions change  the  "sent  time"  in  the  trace  block. 

3.3  Analysis 

To  pinpoint  the  cause  of  the  excessive  processing  delay 
we  shall  use  a model  of  the  nodal  priority  structure.  A 
description  of  that  structure  follows. 

3.3.1  Sysigm 

The  IMPS  are  required  to  perform  a set  of  functions  in 
order  that  the  network  operate  smoothly.  Among  these  func- 
tions are;  receiving/  routing  and  sending  packets^  process- 
ing routing  updates  and  periodically  sending  routing  updates 
to  neighbor  nodes.  A priority  level  is  assigned  to  each  of 
the  various  functions  within  the  IMP.  A partial  list  of  the 
levels  appears  in  Table  3.6.  A function  is  activated  by  an 
interrupt  mechanism.  Only  the  function  associated  with  the 
highest  active  priority  level  may  occupy  the  processor  at  a 
given  instant. 


Abbreviation  Level 


M2i  (highest)  0 


I2M  2 

I2H  3 

H21  4 

T.O  5 

TSK  6 


BCK  (lowest)  7 


Meaning^  Comients 

Modem  to  IMP/  Channel  input 
IMP  to  Modem/  Channel  output 
IMP  to  HOST/  HOST  output 
HOST  to  IMP/  HOST  input 
Timeout/  Periodic  functions 
Task/  Primarily  packet  routing 
Background/  Statistics/  etc. 


Table  3.6.  IMP  priority  levels 


The  portion  of  the  priority  structure  in  which  we  are 
most  interested  here  is  that  governing  the  processing  of 
routing  update  packets  and  other  (non-routing)  packets. 
Under  the  current  scheme/  pictured  in  Figure  3.1(a)/  all 
packets  arrive  and  are  treated  by  the  input  routine  running 
at  the  M2I  level  which  places  them/  without  examination  as 
to  type  or  priority/  on  the  Task  queue.  They  are  then  pro- 
cessed in  f i rst -come-f i rst-served  order.  When  the  Task 
routine  encounters  a routing  packet  it/  in  effect/  switches 
priority  to  level  T.O  so  as  not  to  be  interrupted  by  any 
function  of  equal  or  lower  priority.  (This  has  the  effect 
of  disallowing  the  sending  of  a routing  update  packet  to  a 
neighbor  while  the  table  is  in  the  process  of  being  undat- 
ed.) Non-routing  packets  are/  on  the  other  hand/  interrupt- 
able  by  T.O  functions.  More  detailed  descriptions  appear  in 
CCole  71]  and  in  CMcQu  72]. 
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Figure  3.1(a).  Simplified  model  of  ARPANET  queue  structure. 


i'igure  3.1  (b).  Simplified  model  of  ARPANET  queue  structure  with  priority  within  the  TASK  queue. 


Since^  as  we  see  later/  routing  update  packets  require 
much  processing  compared  with  non-routing  packets/  it  seems 
appropriate  to  reorder  the  priorities  and  treat  routino 
update  packets  in  the  BCK  level  and  other  packets  at  the  TSK 
level.  Figure  3.1(b)  shows  the  logical  structure  of  such  a 
system.  Notice  that  the  Task  queue  has  two  levels  of  prior- 
ity instead  of  one  as  before.  Also  notice  that  unlike  the 
channel/  the  Task  routine  serves  the  queue  by  preempting/  if 
necessary/  the  (low  priority)  routing  updates.  This  scheme 
is  not  new.  Originally  routing  packets  were  treated  in  the 
BCK  level.  The  chief  reason  for  the  current  set  of  priori- 
ties coupled  with  higher  frequency  updating  is  to  propagate 
routing  information  faster  through  the  larger  network 
C Sant  753  . 

3.3.2  Ggnecai  ttlfiQC)' 

This  system  can  be  modeled  by  a single  server  head  of 
the  line  (HOL)  preemptive  priority  queueing  system 
LKlei  763.  The  solution  given  in  CKlei  763  for  the  mean 
time  in  system  for  priority  class  p is  valid  under  the  W/G/1 
assumption  (which  we  later  adopt).  There  are/  however/  some 
partial  results  for  the  G/G/1  case  which  we  may  apply  here. 
Schrage  CSchr  633  proves  that  the  shortest  remaining  pro- 
cessing time  first  (SRPT)  scheduling  algorithm  is  optimal  in 
that  it  minimizes  the  average  number  of  jobs  in  the  system. 
That  is/  the  number  of  jobs  in  a system  under  the  SRPT  algo- 
rithm is  less  than  or  equal  to  the  number  in  the  system 


t 
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under  any  other  scheduling  algorithm  for  the  same  seouence 
of  jobs.  Kleinrock  CKlei  761  points  out  that  if  the  cost  of 
delaying  a customer  is  constant  over  all  priority  classes^ 
then  the  SRPT  rule  achieves  the  minimum  cost.  Hence^  the 
theoretical  results  indicate  that  an  arrival  should  join  the 
queue  behind  all  those  customers  in  the  system  with  remain- 
ing service  time  which  is  less  than  (or  equal  to)  the  ser- 
vice time  required  of  that  arrivals  and  in  front  of  all  oth- 
ers in  the  system.  That  is  all  non-routing  packets  should 
preempts  if  necessary^  the  processing  of  routing  packets^ 
since  as  we  see  below/  routing  packets  require  much  more 
processing  than  do  non-routing  packets. 

Having  stated  the  general  theoretical  result  we  now 
wish  to  show  the  order  of  magnitude  of  the  improvement  one 
may  achieve  by  causing  routing  updates  to  be  processed  at  a 
lower  priority  than  other  packets. 

3.3.3  A ladeL 

In  the  following  analysis  we  wish  to  examine  the  tra- 
deoff between  the  existing  and  the  proposed  system  over  the 
full  range  of  packet  traffic  intensity.  Some  simplifying 
assumptions  will  allow  the  direct  application  of  the  theory. 
For  simplicity/  we  shall  lump  all  high  priority  functions 
into  the  priority  class.  We  consider  two  such  high 
priority  functions  - (a)  modem  input  and  (b)  modem  output; 
and  ignore  all  others.  We  shall  overestimate  the  time  used 
in  these  functions  which  will  tend  to  give  a lower  bound  on 
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performance  (i.e.^ 


a pessimistic  value).  We  assume  that  the 


arrival  of  customers  to  each  class  is  independent  of  the 
arrivals  to  the  other  classes  and  may  be  modeled  by  s Pois- 
son process.  The  Poisson  assumption  on  the  arrival  of  pack- 
ets is  equivalent  to  assuming  that  we  have  a large  number  of 
input  channels  collectively  sending  at  the  same  rate  as  the 
actual  number  of  channels.  We  further  assume  that  each 
queue  has  no  restriction  as  to  its  length  (i.e.r  infinite 
nodal  storage)  and  that  the  overhead  for  changing  ^tasks  is 
negligible.  The  current  system  then^  may  be  modeled  by  a 
two  level  preemptive  resume  HOL  priority  queueing  system  and 
the  proposed  system  by  a three  level  system.  These  systems 
are  pictured  in  Figure  3.2.  In  each  system  the  input/output 
function  has  preemptive  priority  over  the  other  classles). 
The  three  level  system  (in  part  (b)  of  the  figure)  divides 
the  Task  queue  arrivals  into  routing  updates  and  non-routing 
packets^  the  latter  having  preemptive  priority  over  the 
former. 


Fiflurt  3.2.  (b)  Th«  thraa  laval  modal. 


3.3.4  AQaij£ii£  ESSUitS 

The  general  solution  from  TKlei  763  for  the  average 
time  in  system  for  a customer  of  the  pth  class  is 

P 

x(p)  C1-s(p)  3 + r(j)i?(})/2 

3=P 

T(p/’P)  = (3.1) 

t1-s(p)DC1-s(p+1)3 

where  P is  the  number  of  classes  (1  is  the  lowest  priority 
class  and  P is  the  highest)^  x(p)  is  the  mean  service  time 
for  a customer  in  class  p^  x^lp)  is  the  second  moment  of 
service  time  for  a customer  in  class  pr  rlj)  is  the  input 
rate  of  customers  of  class  j/  and  s(p)  is  the  cumulative 
utilization  due  to  classes  of  priority  p and  higher  (where 
the  utilization  for  class  j is  x(j)r(j)).  Let  x(i)  = the 
service  time  for  the  M2I  class^  x(u)  = the  service  time  for 
routing  update  processing/  x<n)  = the  service  time  for  non- 
routing packets/  r(u)  = the  arrival  rate  of  routing  update 
packets/  and  r(n)  = the  arrival  rate  of  non-routing  packets. 
For  the  purpose  of  further  simplifying  the  model  we  shall 
assume  that  the  service  time  in  each  class  is  fixed.  In 
particular/  this  means  that  the  second  moment  is  equal  to 
twice  the  mean.  VJe  assume  that  each  packet  processed  by  the 
Task  routine  arrives  and  eventually  leaves  via  the  m2I  level 
(since  each  packet  processed  by  the  Task  routine  must  arrive 
via  the  input  routine  and  exit  via  the  output  routine  in  our 
model).  We  also  assume  that  the  number  of  routing  update 
packets  received  by  a node  is  equal  to  the  number  sent.  We 


therefore  assume  (an  independent  arrival  process  with)  an 
arrival  rate  of  2Cr(u)+r(n)D  to  the  W?I  class.  From  equa- 
tion 3.1  we  have 


T(1/2)  = 


x(u) r(u)+x(n) r(n) 


Cr<u)+r(n)DC1-2x(i)Cr(u)+r(n)]] 


2 2 2 
Cx  (u>r(u)+x  (n)r<n)]/2  ♦ Cr(u)+r(n)]x  (i) 


C1-x(u) r(u)-x(n) r(n)-2x(i) Cr(u)+r(n)DDC1-2x(i) Cr(u)+r(n) JJ 


(3.2) 


Equation  3.2  aives  the  average  time  in  system  for  routing 
and  non-routing  packets  in  the  two  level  system.  The  solu- 
tion to  the  three  level  system  is 


T(1,3)  = 


1-x<n)r(n)-2x(i)Cr(u)+r(n)D 

2 2 ? 
r(u)x  (u)/2  + r(n)x  (n)/2  + Cr(u)+r(n)Dx  (i) 

1-x(n)r(n)-2x(i) Cr(u)+r(n)D 

1-x(u)r(u)-x(n)r(n)-2x(i)Cr(u)+r(n)3 


(3.3) 


T(2/3)  = 


1-2x (i ) Cr (u)+r (n)D 

2 ? 
r(n)x  (n)/2  ♦ Cr ( u ) ♦r (n ) Dx  (i) 

ri-x(n)r(n)-2x(i)Cr(u)+r(n)3]C1-2x(i)Cr(u)+r(n)3D 


9®^ 


Since  both  systems  are  preemptive^  there  is  no  difference 
between  T<2/2)  and  T<3^3).  Therefore  we  need  only  concern 
ourselves  with  T(1>>2)r  T(2>.3)  and  T(1/3).  T<1/?)  * T(2*-3) 
gives  the  reduction  in  delay  for  non-routing  packets  when 
changing  from  the  two  level  system  to  the  three  level  sys- 
tem- T{1/3)  - T(1^2)  gives  the  increase  for  routing  update 
processino  for  the  same  change.  Comparison  of  the  two  sys- 
tems depends  on  the  relative  values  of  x(u)^  x(n)^  r(u)  and 
r<n)  which  we  shall  now  estimate  in  order  to  present  some 
numerical  results. 

3.3.5  Nyffigcisai  CSSWiU 

We  estimate  that  no  more  than  300  machine  cycles  are 
required  to  process  an  incoming  packet  or  an  outooing  pack- 
et. We  further  assume  that  a maximum  of  300  cycles  are 
required  to  do  the  TSK  level  processing  of  a non-routing 
packet.  These  values  are  purposely  larger  than  the  estimate 
in  CKcQu  723.  This  is  in  order  to  exaggerate  their  effect 
in  the  model  and  therefore  provide  a pessimistic  projection 
of  performance  (as  mentioned  earlier).  Routing  packets 
require  much  more  time  to  digest.  In  a recent  experiment 
carried  out  BBN  CB6N  753  with  a 316  IMP#  it  was  found  that 
IPX  of  the  HOST  throughput  was  lost  when  going  from  zero  to 
one  50kbps  line.  A similar  loss  was  noted  with  each  addi- 
tional line.  Assuming  that  the  processor  was  fully  utilized 
durina  the  experiment#  this  indicates  that  routing  and  other 
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processes  associated  with  IMP-to-IMP  lines  require  14400 
cycles/routing  update  period.  The  time  could  be  signifi- 
cantly reduced  by  implementing  the  "local  loop"  prevention 
algorithm  outlined  in  iNayl  75D  and  chapter  2.  That  iS/’ 
many  (one  third  on  the  average  for  3-connected  neighbors)  of 
the  routing  update  entries  are  skipped  since  they  point 
toward  the  updating  node.  We  estimate  those  orocesses  in- 
volved in  handling  one  line  (with  no  data  traffic)  as  fol- 
lows: 


Function 


Cycles/update  period 


Receive  routing  message 

Input  (140  on  316)  280 

M2I  300 

TSK  300 

Receive  and  process  IHY  100 

Send  routing  packet  300 

T.O  300 

I2M  300 

Output  (140  on  316)  280 

Process  routing  packet  12000 

Other  540 


Total  14400 


Table  3.7  Channel  function  processing  time  estimates 


We  shall  use  for  x(u)  a value  of  12000^  which  is  a con- 
servative estimate  to  minimize  the  effect  of  routing.  this 
conflicts  with  the  pessimistic  estimates  of  inputs  output 
and  non-routing  packet  processing  and  would  tend  to  give  an 
optimistic  value  of  performance  (i.e./  lower  than  actual 
delay).  However^  it  does  result  in  a lower  bound  (or  pes- 
simistic estimate)  for  the  improvement  in  performance 


72 


between  the  two  systems.  The  service  time  of  the  various 
tasks  is  a function  of  the  processor  speed.  In  August  1976 
there  were  four  processor  types  in  the  ARPANET  — the  516 
IMP/  316  IMP/  316  TIP/  and  Pluribus  [Hear  763  IMP  according 
to  CNIC  763.  We  shall  consider  only  the  first  three  which 
have  speeds  of  .96/  1.6  and  2.2  usec/cycle  respectively 
CMcQu  733.  The  rate  of  arrival  of  routing  packets  to  a node 
is  governed  both  by  the  number  of  connected  channels/  and 
the  utilization  of  those  channels  as  mentioned  in  chapter  2. 
The  three  nodal  types  were  distributed  with  connectivity 
CNIC  763  as  shown  in  Table  3.R.  Also  shown  in  Table  3.P  is 
the  predicted  "zero-load”  average  waiting  time  expressed  in 
milliseconds  for  each  case  (i.e./  the  waiting  time  when  r(n) 
= 0).  Each  pair  in  the  table  consists  of  the  number  of 
nodes  in  the  network  with  this  connectivity  followed  by  the 
zero  load  delay  prediction.  At  that  time  there  was  one 
five-connected  Pluribus  IMP  in  the  network. 


Connect ivity 

516 

316 

TIP 

1 

0/  .63 

2/ 

1 .87 

3/ 

3.79 

2 

5/  1.40 

9/ 

4.57 

12/ 

10.38 

3 

11/  2.36 

5/ 

8.83 

9/ 

24.93 

4 

1/  3.61 

1 / 

16.66 

0/ 

86.17 

Table  3.8  Connectivity  of  processor  types 

The  zero  load  average  waiting  time  for  non-routing  packets 
in  the  three  level  system  is  essentially  zero.  The  range  is 
from  .000645  to  .0130  msec.  Comparing  these  values  with 


those  in  Table  3.8  we  see  the  dramatic  effect  of  this  change 
in  priorities.  The  zero  Load  average  waiting  time  for  non- 
routing packets  in  the  two  level  system  is  from  aoout  1000 
to  6000  times  what  it  is  in  the  three  level  system! 

The  *^ehavior  of  these  systems  over  the  complete  range 
of  channel  utilization  (where  packets  of  300  bits  average 
Length  occupy  the  channels)  is  shown  in  Figure  3.3  parts  (a) 
through  (f).  Also  shown  in  the  figure  are  measured  data 
points  (as  squares)  where  such  data  exists.  We  show  only 
the  results  for  the  two  and  three  connected  nodes  (those 
which  are  most  prevalent  in  the  ARPANET).  Each  part  of  Fig- 
ure 3.3  consists  of  three  curves.  The  curve  which  lies 
between  the  two  others  shows  the  performance  of  the  two  lev- 
el system  and  the  other  two  show  the  three  Level  system's 
performance.  Average  waiting  time  for  routing  and  non- 
routing packets  is  shown  in  the  highest  and  lowest  curves 
respectively.  The  gain  in  delay  for  non-routing  packets  is 
the  difference  between  the  middle  and  the  lowest  curve.  The 
cost  in  delay  for  routing  is  the  difference  between  the 
highest  and  the  middle  curve.  For  parts  (a)  and  (b)  the 
bottom  curve  is  almost  indistinguishable  from  the  axis 
(i.e.^  nearly  zero  waiting  time  through  the  entire  range). 
Notice  in  parts  (b)  and  (d)  that  the  model  is  a good  lower 
bound  to  the  measured  results.  Since  we  overestimated  the 
high  level  processing  time^  an  upper  bound  was  expected. 
The  fact  that  a lower  bound  was  achieved  is  most  likely  due 
to  the  underestimation  of  x(u). 


W (msMs) 


Channal  utilization 


Fifura  3.3.  (bi  Moan  waiting  timo  on  TASK  queue  (3-connected  516  IMP). 
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Figure  3.3.  (c)  Mean  waiting  time  on  TASK  queue  (2-connected  316  IMP). 


77 


»0-AUH7  •♦96  CALIFORNIA  UNIV  LOS  ANGELES  DEPT  OF  COMPUTER  SCIENCE  F/6  17/2 

ADVANCED  TELEPROCESSING  SYSTEMS. (U) 

JUN  77  L KLEINROCKr  G ESTRIN#  R MUNTZ  DAHC15-73-C-0368 

UNCLASSIFIED  mi 


2 Of  3 

- 

^^047496 

"■ , 

k,..— 

9 

IBM 

■ 

K\ 

■ - 

J J. . 

r "■ 

h 

\ - 

' -T'-. 

. 

!X  N 

1 \ 

- 

’ 

— 

— 

J * 

L 

- 

' 1 

, 

^ 

- 

.. 

-r" 

-- 

* 

- 

v. 

\ 

y 

-f?" 

----- 

V.  . 

K. 

t 

1 

. . X . . 

S 

« 

1 

_ . 

V 

1 

. V . , 

i 

s 

■ ■ ' 

1 

1 

0 0.2  0.4  0.6  0.8  1.0 

Channel  utilization 


Figura  3.3.  (d)  Maan  waiting  tinw  on  T ASK  queua  (3-connected  316  IMP). 


78 


fl 

'I 

I 

Substituting  the  two  level  system  model  for  the  con- 
stant processing  delay  term  i the  CKlei  74D  model  causes 
the  predicted  delay  to  increase  to  155  msec  for  March  1977. 

This  is  an  increase  of  126X  (i.e./  from  73  to  165  msec)  over 
the  August  1973  value  of  predicted  delay.  Recall  that  the 
^ actual  increase  in  delay  was  145X  (i.e.#  form  93  to  22S 

msec).  A better  (i.e.#  larger)  estimate  for  xCu)  would 
drive  this  prediction  closer  to  the  actual  delay.  Assuming 
that  the  ratio  of  predicted  delay  to  actual  (measured)  delay 
remains  fixed  (i.e.^  73/93)/  then  the  projected  mean  round- 
trip  delay  for  March  1977  would  be  approximately  129  msec  if 
I routing  updates  were  processed  at  low  priority.  This 

represents  a 44%  reduction  in  delay! 

It  is  clear  from  these  results  that  a substantial 
j reduction  in  nodal  processing  delay  would  result  from  reord- 

ering the  priorities  within  the  system.  8ut  this  is  a local 
optimization  which  may  eventually  lead  to  worse  overall  per- 
formance in  a network-wide  sense.  Because  the  speed  of  the 

processing  of  routing  information  is  substantially  reduced  ^ 

, 

by  the  reordering/  it  may  happen  that  routing  information 
will  not  be  propagated  as  quickly  as  needed.  Looking  at  the 
three-connected  TIP  for  example/  we  see  that  routine,  pro- 
cessing incurs  infinite  delay  at  about  .6  channel  utiliza- 
tion. To  determine  whether  this  local  optimization  leads  to 
a global  optimum  or  leads  instead  to  disaster/  let  us  exam- 
ine the  simulation  results  in  the  next  section. 
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3.4  siBuialiaa 

A simulation  was  performed  under  the  same  conditions  as 
described  in  chapter  2.  The  local  loop  free  algorithm  was 
modified  to  perform  routing  processing  at  low  priority. 
This  new  algorithm  (LLFLR)  was  used  in  the  simulation.  The 
result  is  compared  in  Figure  3.4  to  the  LLFR  algorithm  of 
chapter  2.  The  figure  shows  the  network-wide  mean  delay  as 
a function  of  offered  load.  It  appears  that  the  local  op- 
timization in  this  case  leads  to  a global  optimum#  since 
LLFLR  out  performs  LLFR  at  each  load.  One  would  therefore 
expect  that  if  low  priority  routing  update  processing  were 
done  in  the  ARPANET#  then  the  mean  round-trip  delay  woul"^ 
fall  much  closer  to  the  predicted  (rather  than  the  measured) 
values  in  Table  3.1  for  flay  1974  and  August  1977#  as  indi- 
cated in  the  last  section.  This  would  reduce  the  mean  delay 
to  be  within  the  specification  of  200  msec. 


3.5  CQQSiUSiQQ^ 

There  are  two  important  general  Lessons  to  be  learned 
here.  (1)  In  order  to  provide  different  levels  of  service 
to  different  classes  of  customers^  one  must  be  careful  to 
provide  that  order  of  service  at  each  step  of  the  process- 
ing. For  example^  if  one  wishes  to  guarantee  that  priority 
packets  traverse  the  network  more  quickly/  on  the  average/ 
than  do  non-priority  packets/  then  priority  packets  must  be 
serviced  with  priority  from  queues  in  which  they  may 
wait  (i.e./  the  Task  queue  as  well  as  the  channel  queue/  in 
this  case).  ii.)  Examination  of  the  worst  case  phenomenon 
should  not  be  used  alone  in  arriving  at  decisions  regarding 
priority  of  service.  Rather  one  should  evaluate  the  impli- 
cation of  those  decisions  over  the  entire  range  of  system 
oper  at  ion . 

We  have  concentrated  in  this  chapter  on  a small  (but 
important)  part  of  the  priority  structure  within  the  ARPANET 
nodes.  There  are  other  areas  within  the  system  priority 
structure  which  may  bear  fruit  in  terms  of  reducing  delay  or 
increasing  throughput.  There  is  a priority  structure  with 
which  the  channels  are  accessed  as  well/  which  provides 
another  area  for  investigation. 

In  this  chapter  a high  degree  of  overhead  due  to  the 
ARPANET  routing  update  procedure  was  exposed.  The  next 
chapter  concentrates  on  the  issue  of  overhead  in  periodic 
adaptive  routing  in  large  networks. 


CHAPTER  A 

ON  THE  EFFECT  OF  PERIODIC  UPDATE  ROUTING  PROCEDURES 


^•1  lQirQdu£ti&Q 

In  a packet  snitched  networks  some  form  of  adaptive 
routinq  procedure  is  desirable  so  that  packets  may  be  routed 
around  Line  and  node  failures  and  possibly  around  congestion 
in  the  netnork  and  to  allow  the  network  to  adjust  to  dras- 
tics changes  in  input  traffic  matrix.  It  is  clear  that 
there  must  be  some  overhead  associated  with  any  form  of 
adaptive  routing  (i.e.^  the  channel  time  and  processor  time 
required  to  generate#  transmit#  and  process  the  routinq 
information).  Clearly#  one  would  hope  that  the  cost  for 
such  adaptive  routing  does  not  exceed  the  benefits  derived 
therefrom.  Since  adaptive  routing  is  considered  to  be 
necessary  in  practice#  its  overhead  has  received  only  par- 
tial consideration  by  most  authors  CCegr  753#  CFult  723# 
CMcCo  753#  CMcQu  743#  CPick  763.  In  this  chapter#  we  study 
some  unusual  phenomena  caused  by  the  interference  of  routinq 
updates.  Specifically#  we  consider  the  cost  (in  terms  of 
message  delay)  of  the  current  ARPANET  routing  update  pro- 
cedure. We  begin  by  presenting  some  results  of  a set  of 
measurement  experiments  which  prompted  an  analysis  of  the 


* This  chapter  is  a revised  version  of  CNavl  763 
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effects  of  the  routing  procedure  on  message  delay.  ^ sirr- 
plified  model  of  the  system  is  then  discussed  and  analyzed 
exactly.  This  exact  solution  is  a bit  unwieldy  for  highly 
detailed  models  in  which  case  we  resort  to  simulation  to 
show  the  performance  and  to  demonstrate  the  effect  of  modi- 
fied routing  schemes.  The  simulation  results  indicate  a 
rather  high  cost  associated  with  the  use  of  periodic  routing 
updates  in  networks  the  size  of  the  ARPANET.  This  suggests 
the  use  o "passive"  routing  scheme  with  catastrophe- 
triggered 

4.2  HJgasycgmgQi 

»le  set  out  to  determine  by  what  means  and  how  accurate- 
ly one  could  predict  round-trip  network  delay  for  a stream 
of  messages  with  fixed  interarrival  times  based  on  previous 
delay  samples  in  the  ARPANET.  This  traffic  pattern  is  exhi- 
bited by  fixed  data  rate  sources  such  as  soeech  CForo  75aD. 
These  measurement  experiments  were  conducted  with  no  inten- 
tion of  considering  the  effects  of  the  oer’odic  update 
scheme  used  in  the  ARPANET.  We  observed  a much  lower  than 
expected  correlation  between  successive  delays  (see  Chapter 
5).  In  experiments  sending  data  as  fast  as  possible/  suc- 
cessive delays  display  higher  correlation  CKlei  7Sa]  than  do 
those  for  a fixed  interarrival  time  source,  a closer  look 
(suggested  by  0.  Cohen  of  the  Information  Sciences 
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Institute^  University  of  Southern  California)  revealed  some 
interesting  phenomena  regarding  the  effect  of  periodic  rout- 
ing update  procedures. 

The  experiments  took  place  on  Friday  evening  December 
12/  1975/  between  the  hours  of  9 and  11  p.m.  PST.  (A  light 
network  load  and  therefore  nearly  constant  delay  was  expect- 
ed during  this  time  period.)  Full  single-packet  messages 
were  sent  from  the  UCLA  PDP  11/45  to  a "discard  fake  HOST" 
(a  portion  of  the  ARPANET  IMP  software  which  mimics  a "real 
HOST"  acting  as  a sink  for  a message  stream/  see  CBBN  69]) 
over  (minimum  hop  path)  distances  of  1/  2/  5 and  10  hops  at 
fixed  interdeparture  times  of  124/  165/  and  248  msec.  The 
round-trip  delay  (i.e./  the  delay  from  the  time  the  message 
is  ready  to  be  sent  until  the  end-to-end  acknowledgement  — 
RFNM  — is  returned)  was  measured  by  the  PDP  11/45  and 
recorded  for  subsequent  study. 

Figures  4.1  through  4.4  show  some  of  the  round-trip 
delay  measurements  plotted  against  message  sequence  number 
(i.e./  time).  Here  we  show  network  delay  as  a function  of 
(message  arrival)  time.  These  particular  samples  are 
representative  of  the  collection  of  experiments. 

Notice  the  unusual  increases  in  delay  at  regular  inter- 
vals (of  about  30  messages)  in  Figure  4.1.  In  each  interval 
there  is  a group  of  three  dominant  peaks  separated  by  two 

regularly  spaced  points  where  the  delay  is  near  the  minimum. 

< 

Notice  also  the  regular  decrease  in  the  first  peak  in  each 
group  with  time  until  it  is  replaced  with  a full  sized  peak 
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at  intervals  of  five  groups.  This  curve  is  in  fact  a 
periodic  function  (with  some  "noise"  due  to  the  "other" 
backgrouno  data  traffic)  with  a period  of  about  15C  mes- 
sages! (Our  model  in  a later  section  and  the  correlation 
results  in  Chapter  5 show  that  the  period  is  actually  160.) 

This  periodic  behavior  is  Less  noticeable  at  longer 
network  distances.  There  is  a pattern  of  climbing  to  a 
local  maximum  and  suddenly  dropping  and  starting  the  climb 
again  in  Figures  4.2  and  4.3.  Generally  speaking  increases 
in  delay  are  gradual  while  decreases  are  immediate  (though 
the  opposite  condition  occasionally  occurs  as  well).  The 
curve  shown  in  Figure  4.4  varies  quite  severely.  There  is  a 
pattern  however^  and  very  close  examination  reveals  a 
periodic  function  (again  with  some  noise)  with  a period  of 
approximately  130  messages.  Plots  of  other  samples  show 
that  the  shape  of  the  curves  is  more  related  to  the  data 
rate  than  to  network  distance. 

One  would  not  normally  expect  to  see  such  a regular 
variation  in  delay  with  time  assuming  no  control  on  the  oth- 
er network  traffic.  Rather  one  might  expect  to  see  a random 
function  of  time#  possibly  with  a slowly  varying  average 
which  changes  with  network  load  CCohe  74],  (Indeed#  the 
latter  was  our  hope#  we  were  looking  for  delay  predictors.) 
Below  we  present  an  analysis  of  the  regularity  of  these 
delay  functions. 
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Sefore  proceeding  with  the  analysis^  we  must  first 
describe  the  important  features  of  the  periodic  routino 
update  scheme  in  the  ARPANET. 

4.3.1  eSLiQdi£  CauilQa  ichemg  d£S.£LiCtiQQ 

There  exists  a basic  routing  interval  of  640  msec.  The 
beginning  time  for  the  basic  period  is  chosen  essentially  at 
random  for  each  half-duplex  channel  in  the  network.  It  was 
noted^  by  BBN/  that  as  the  network  grew  in  size/  routino 
information  was  not  being  propagated  in  a timely  fashion 
with  only  one  update  per  basic  period  CBPN  743.  Therefore 
provision  was  made  to  send  up  to  five  updates  in  one  basic 
period.  During  a basic  period  the  line  utilization  (includ- 
ing that  for  updates)  is  measured  to  determine  the  number  of 
updates  to  be  sent  during  the  next  basic  period.  For  each 
additional  2(iX  of  line  utilization/  one  of  the  five  possible 
updates  is  dropped.  For  example/  at  65X  line  utilization 
only  two  updates  are  sent  in  the  next  basic  period.  It  is 
important  to  note  that  the  updates  are  not  necessarily  even- 
ly spaced  within  the  basic  period.  Rather  this  period  is 
divided  into  five  equal  segments.  Routing  updates  are  sent 
only  at  segment  boundaries  (i.e./  every  128  msec).  For  the 
2LI  to  40%  range/  for  example/  updates  are  sent  at  C/  128/ 
384/  and  512  msec  into  the  basic  period  (rather  than  0/  160/ 


) 32U/  480  msec  for  evenly  spaced  updates).  Routing  update 

t packets  were  1160  bits  in  length/  requiring  23.2  msec  to 


transmit  on  the  5C  kbps  channels  used  in  the  ARPANET  (recent 
changes  have  increased  this  somewhat).  Additional ly^  ap- 
proximately 12000  machine  cycles  (based  on  a measurement 
reported  in  CBBN  75D^  see  Chapter  3)  are  required  to  digest 
an  incoming  routing  update  (i.e.#  11.52  msec  for  a 516  IMP 
and  19.2  msec  for  a 316  IMP).  Chapter  3 examines  the  impor- 
tance of  this  overhead  within  the  nodes. 

4.3.2  A simete  mgdSlk 

Cur  analysis  uses  a queueing  system  with  some  "back- 
ground" traffic  (i.e.r  routing  and  ambient  data  traffic)  to 
which  we  add  a stream  of  deterministically  generated  traf- 
fic. The  inclusion  of  the  ambient  data  traffic  is  to  model 
the  interference  caused  by  other  packet  sources  in  the  sys- 
tem. We  wish  to  study  the  system  time  of  this  added  stream 
traffic  (i.e.^  the  round-trip  delay  as  shown  in  Figures  4.1 
through  4.4).  We  first  examine  the  waiting  time  on  a single 
channel.  The  model  for  the  single  channel  is  pictured  in 
Figure  4.5.  There  are  three  classes  of  customers  arriving 
to  a single  queue;  routing  update  packets  (R)/  ambient  data 


packets  (A)/  and  stream  packets  (S).  All  arrivals  are 
served  in  f i rst-come-f i r st-served  (FCFS)  fashion  (thouph  in 
the  actual  system^  routing  update  packets  take  precedence 
over  both  priority  data  packets  and  control  packets/’  all  of 
which  take  precedence  over  non-priority  data  packets;  see 
Chapter  3).  The  effect  of  this  is  negligible  for  our  pur- 
pose here  as  we  show  later. 

Let  w<t)  represent  the  waiting  time  for  a stream  traf- 
fic packet  arriving  at  time  t.  Suppose  U(t)  is  the  amount 
of  work  of  type  R and  A remaining  in  the  system  at  time  t. 
As  long  as  the  stream  arrivals  have  no  priority  over  other 
customers^  an  arrival  must  wait  until  all  work  in  the  system 
has  been  completed  before  receiving  any  service  of  its  own. 
The  amount  of  work  in  the  system  found  by  an  arrival  at  time 
t is  at  least  U<t).  Hence 

u(t)  > U(t) 

Equality  is  achieved  for  an  arrival  when  it  is  the  first 
stream  traffic  arrival  in  a busy  period  (i.e.#  a period  of 
continuous  activity  by  the  server). 

For  simplicity  of  the  following  analysis/  we  assume 
that  the  ambient  data  traffic  has  zero  intensity.  U(t)  for 
a system  void  of  any  data  traffic  is  shown  in  Figure  4.6. 
At  the  arrival  time  of  a routina  update  packet/  the  amount 
of  work  in  the  system  jumps  up  by  23.2  msec  (the  service 
time  of  a routing  update  packet).  With  no  other  packets  in 
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the  system^  the  routinu  uodate  packet  is  i mipedi ate ly  served 
at  a rate  of  one  second  per  seconds  and  exits  the  channel 
after  23.2  msec.  After  the  departure  there  is  nc  work  in 
the  system  until  the  next  arrival. 

A more  interesting  measure  of  performance  is  w(sn)^  the 
waiting  time  for  the  nth  message^  where  the  constant  in- 
terarrival time  of  the  stream  traffic  is  s.  Figure  A. 7 is  a 
plot  of  U(sn)  for  s = 248/.  165^  and  124  msec  (i.e.^  the 
periods  used  in  the  measurement  experiments).  For  these 
values  of  s (and  the  message  size  used  in  the  measurement 
experiments)  the  line  utilization  is  in  the  20  to  40%  range 
so  that  every  fifth  update  is  dropped.  These  single  channel 
curves  nicely  display  some  of  the  characteristics  of  delay 
shown  in  Figures  4.1  through  4.4.  The  major  shape  of  Figure 
4.1  is  nearly  the  same  as  U(124n).  U(165n)  and  U(24Rn) 
display  the  relative  variation  of  the  previous  corresponding 
figures.  That  is  U(165n)  varies  more  rapidly  than  does 
U(248n)  which  in  turn  varies  more  rapidly  than  U(124n);  and 
correspondingly  Figure  4.4  varies  more  rapidly  than  do  Fig- 
ures 4.2  and  4.3  which  in  turn  vary  more  rapidly  than  Figure 
4.1.  One  can  clearly  identify  the  period  of  each  U(sn) 
curve.  Indeed^  let  PCs)  represent  the  period  of  U(sn). 


Then 
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P(s)  = LCM(s/r)/s 

where  LCM(x/y)  = the  least  common  multiple  of  x and  yr  and  r 
= the  routing  period  (i.e.^  the  minimum  time  r such  that  the 
pattern  of  routing  arrivals  is  the  same  in  the  intervals 
(C/r)/  (r#2r)/  (2r^3r)/  ...).  For  these  examples^  r=640 
msec.  P(s)  has  the  values  80/  128/  160  for  s = 24P/  165/ 
and  124  respectively. 

Figure  4.3  shows  the  detailed  precession  of  the  periods 
of  the  routing  and  stream  traffic.  Each  row  in  Figure  4.8 
shows  one  period  of  the  stream  data.  The  relative  positions 
of  the  data  (S)  packets  and  the  routing  (R)  packets  within 
the  period  are  shown.  Notice  that  every  fifth  routing  up- 
date packet  is  missing.  The  figure  makes  clear  the  fact 
that  the  priority  which  routing  update  packets  carry  has  no 


effect  (in  the  absence  of  ambient  data  traffic).  That  is/ 
data  packets  are  delayed  (by  routing  update  packets)  only 
when  they  arrive  while  the  channel  is  busy  serving  a routing 
update  packet.  Since  preemption  is  not  permitted  on  the 
channels/  occasionally  a routing  update  packet  must  wait  for 
a data  packet  to  complete  service  on  the  channel  before 


being  transmitted. 


CHANNEL  1 

_ 1 


CHANNEL  2 
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When  packets  pass  throuqh  more  than  one  tandem  channels 
a more  complicated  pattern  arises.  An  example  pattern  for  a 
system  of  two  tandem  channels  is  shown  in  Figure  4. *5.  Once 
the  packets  are  delayed  at  channel  lx  their  arrival  rate  at 
channel  2 then  corresponds  to  that  of  the  routing  update 
packets.  During  this  time  we  have  two  deterministic  streams 
arriving  at  a constant  offset.  Hence  the  waiting  time  at 
channel  2 remains  constant  (except  for  routing  drop  outs) 
until  the  data  packets  are  no  longer  delayed  at  channel  1. 
This  example  illustrates  the  climbing-dropping  phenomenon 
shown  in  Figures  4.2  and  4.3. 

An  exact  solution  of  the  n-node  tandem  server  model  is 
highly  dependent  on  the  phasing  of  the  routing  updates  on 
the  various  channels^  as  seen  in  Figure  4.9.  This  dependen- 
cy produces  an  unwieldy  solution.  Thereforex  in  the  next 
sectionx  we  use  simulation  to  study  a model  which  is  more 
closely  tied  to  the  actual  system. 

4.4  ^iiEUiatiQD 

Having  shown  the  exact  behavior  for  the  single  channel 
systemx  we  now  resort  to  simulation  to  illustrate  some 
phenomena  present  in  the  more  complicated  tandem  system. 
Essentially  identical  experimentsx  as  described  in  the  above 
measurement  sectionx  were  performed  using  a rather  detailed 
simulation  of  the  ARPANET.  The  simulation  program  was  writ- 
ten in  PL/.I  for  the  IBM  360/91  at  UCLA.  There  were  three 
main  differences  between  the  measurement  and  simulation.  In 


the  simulation#  the  ambient  data  traffic  was  set  to  zero# 
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node-to-node  acknowledgements  were  not  used#  and  the  phasing 
of  the  routing  update  packets  on  the  various  channels  was 
not  synchronized  with  that  of  the  measurement  experiments. 
Therefore#  we  expect  to  see  the  same  period  displayed  but 
possibly  a different  shape  for  the  delay  curves.  Indeed 
Figures  4.1  and  4.10  show  a curve  with  a period  of  320  mes- 
sages. Notice  that  the  period  differs  from  that  of  U(sn) 
for  this  data  rate.  This  is  due  to  the  fact  that  while  the 
data  rate  causes  a stable  rate  of  four  routing  updates  per 
basic  period#  the  return  path  carry ing  RFNM's  alternates 
between  four  and  five  updates  per  basic  period.  This 
results  in  a stable  rate  of  nine  updates  in  each  two  basic 
periods.  That  is#  1280  is  the  minimum  time  r for  which  the 
pattern  of  routing  arrivals  is  fixed  in  the  intervals  (0#r)# 
(r#2r)#  (2r#3r)#...  . Therefore#  the  round-trip  delay  curve 
has  a period  of  L CM( 1 24# 1280) / 1 24  = 320  messages.  Figure 
4.1  may  exhibit  a period  of  160  messages  due  to  the  node- 
to-node  acknowledgements  which  elevate  the  traffic  on  the 
backward  channel  just  enough  to  force  a constant  rate  of 
four  updates  per  basic  period.  One  also  notices  that  the 
minimum  values  for  Figures  4.1  and  4.10  are  not  the  same. 
This  is  due  to  several  factors.  In  the  simulation  we  have 
estimated  the  channel  propagation  time  and  the  nodal  pro- 
cessing time;  both  are  likely  lower  than  their  actual  value. 
Another  factor  is  that  we  have  assumed  zero  acceptance  time 
for  the  message  at  the  destination  node  and  for  the  PFNM  at 
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the  source  node.  These  assumptions  drive  the  delay  down  for 


i 


I 

t 

f 

i 

I 

I 


\ 

\ 

[ 


the  simulation  and  hence  lowers  the  vertical  offset. 

Figures  4.2  and  4.3  are  too  random  to  identify  a 
period.  However#  one  should  notice  the  slow  climbing  and 
rapid  falling  in  the  curves  in  Figures  4.2#  4.3#  4.11  and 

4.12  (though  the  measured  data  is  quite  noisy).  The  most 
rapid  variation#  both  for  simulation  and  measurement#  is  the 
packet  rate  of  approximately  six  per  second  shown  in  Figures 

4.13  and  4.4  respectively.  Roth  curves  possess  a period  of 
128  messages. 

Nestled  between  the  large  delays  in  Figures  4.1  and 
4.10  are  some  small  mounds.  These  are  due  to  the  interfer- 
ence between  routing  updates  and  stream  traffic  in  the 
"TASK"  queue  LMcQu  72D  (i.e.#  data  packets  waiting  to  be 
placed  on  an  output  queue  and  routing  update  packets  waiting 
and  being  digested  into  the  local  routing  table).  The  TASK 
queue  is  currently  served  in  FCFS  fashion.  A conceptually 
simple  modification  is  to  serve  routing  update  packets  at 
low  priority  from  the  TASK  queue.  (This  is  discussed  at 
length  in  Chapter  3.)  Fioures  4.14  through  4.17  show  the 
effect  of  sending  routing  updates  at  the  same  rate  as  be- 
fore# but  processing  data  packets  by  preempting  the  process- 
ing of  interfering  routing  updates.  Notice  the  decrease  in 
both  the  average  and  variance  of  delay.  These  curves  show 
the  best  possible  delay  under  the  current  periodic  update 
scheme#  assuming  preemption  is  not  allowed  on  the  channels 
as  well. 
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For  sending  stream  traffic  in  an  otherwise  empty  net- 
works the  processing  of  routing  updates  at  low  priority  pro- 
vides superior  performance  to  the  current  FCFS  approach. 
There  may  be  some  question  whether  this  superiority  would 
remain  in  a loaded  network.  As  the  traffic  increases^  the 
routing  updates  remain  on  the  TASK  queue  for  a longer  time 
tnus  causing  a possible  laq  in  the  propagation  of  recent 
routing  information.  Eventually/  one  might  expect  that  lag 
would  result  in  worse  routes  being  used/  and  correspondingly 
increased  delays.  This  possibility  must  be  compared  against 
the  added  delay  to  data  traffic  in  the  FCFS  case.  Analysis 
of  a single  node  in  isolation  as  well  as  the  results  of  a 
simulation  of  a network  appear  in  Chapter  3/  with  the  con- 
clusion that  routing  update  processing  at  low  priority  is 
good  both  locally  and  globally. 

One  may  go  one  step  further  and  consider  the  perform- 
ance of  a system  where  routing  update  packets  may  be 
preempted  from  service  on  the  channels  as  well.  This  "total 
background"  routing  may  be  difficult  to  implement  in  prac- 
tice. The  next  step/  if  we  ignore  'letwork  component 
failures  for  the  moment/  is  to  eliminate  routing  updates  and 
to  use  some  form  of  fixed  routing.  As  with  total  background 
routing/  this  causes  no  interference  of  data  packets  due  to 
routing  information  processing.  One  wonders  if  fixed  rout- 
ing will  be  less  effective  since  it  cannot  adapt  tc  changing 
traffic  patterns  (i.e./  at  very  low  traffic  levels  fixed 
routing  performs  better  than  adaptive  schemes/  hut  perhaps 
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may  fail  at  higner  traffic  levels).  However/^  ve  do  recall 
from  CKlei  6A3  that  properly  designed  fixed  routing  pro- 
cedures may  be  superior  to  adaptive  ones.  This  is  further 
supported  by  iPric  723.  Figure  A.I*'  also  s^ows  the  relative 
performance  of  a fixed  scheme  (F»)  and  the  tuo  others  (HPR 
and  LLFLR  as  described  in  Chapters  2 and  3).  To  give  per- 
spective here/  we  note  that  a relative  traffic  load  of  1.0 
is  slightly  higher  than  the  weekly  average  traffic  level 
reported  in  CKlei  743.  we  note  also  that  with  a high  degree 
of  confidence/  the  values  for  mean  delay  are  correct  to 
within  +7X  for  the  first  three  levels  of  relative  traffic 
intensity  (i.e./  1.0/  5.0/  and  10.0).  The  values  above 
intensity  10.0  are  less  precise/  but  the  only  Questionable 
points  are  for  LLFLR  and  FR  at  load  14.  The  range  of  values 
for  LLFLR  ana  FR  overlapped  in  the  several  simulations  for 
this  intensity.  The  fixed  routes  used  were  the  shortest  hop 
paths.  One  could  choose  an  even  better  fixed  routing  scheme 
by  using  the  flow  deviation  method  CGerl  733  for  example. 
Notice  that  fixed  routing  always  performed  better  than  the 
foreground  routing  (HDR)/  and  only  at  very  high  traffic  lev- 
els is  it  worse  than  the  background  processing  routing 
(LLFLR).  This  suggests  that  the  cost  of  routing  in  the 
ARPANET  is  extremely  high  indeed/  since  traffic  levels  are 
currently  very  low. 


We  have  so  far  ignored  network  failures.  It  is  clear 
that  failures  do  occur  in  practice.  Long  term  monitoring  of 
the  ARPANET  tNcke  74J  shows  a mean  time  between  failures 
<MTBF)  of  431  hours  for  lines  and  221  hours  for  nodes. 
Failures  cause  topological  changes  to  occur  in  the  network 
in  the  following  two  ways.  When  a channel  fails/  it  is  as 
if  it  were  removed  from  the  network.  When  a node  fails/  all 
its  attached  channels  are  removed  from  the  network.  We 
define  the  network-wide  MT8F  to  be  the  mean  time  between 
channel  removals.  Then  with  the  S7  nodes  and  65  full  duplex 
channels/  in  the  June  1975  ARPANET/  and  assuming  that  each 
node  is  of  average  connectivity  (i.e./  approximately  2.24 
CBDN  753)/  these  figures  yield  a network-wide  KTBF  of  3.76 
hours . 

A far  better  method  of  routing/  it  appears/  would  be  to 
use  a "passive"  scheme.  In  such  a scheme  one  establishes 
routes  and  continues  to  use  them  in  a fixed  routing  fashion 
until  some  catastrophe  occurs  (i.e./  a failure  or  possibly 
even  severe  congestion).  At  the  time  a catastrophe  occurs 
one  could  "turn  on"  routing  updates  until  the  tables  "sta- 
bilize". From  the  above  data  one  can  see  that  the  average 
time  between  turning  on  routing  due  to  failure  would  be 
almost  two  hours  (or  approximately  10000  times  the  basic 
rout ing  period) ! 

Recently  a technique  called  "explicit  path  routing”  was 
introduced  by  Jueneman  and  Kerr  CJuen  763.  This  scheme  pro- 
vides for  a set  of  fixed  paths  between  any  node  pair  and 


requires  no  updating  of  routing  tables.  This  procedure 
deserves  further  investigation  in  light  of  our  results. 

4.5  CQQ£iUSlQQS 

In  this  chapter  we  have  shown  some  interesting  message 
delay  phenomena  for  periodic  stream  traffic  attributable  to 
the  periodic  routing  scheme  used  in  the  ARPANET.  We  con- 
clude that  periodic  routing  is  quite  costly  in  medium  sizef^ 
(and  bigger)  networks.  In  order  to  assure  good  performance 
in  the  face  of  failure  (or  heavy  congestion)  one  pays  ? high 
price  in  terms  of  message  delay  for  periodic  routing  update 
procedures  in  networks  of  the  size  of  the  ARPANET.  we  sug- 
gest the  use  of  a passive  routing  scheme^  in  which  updates 
are  scheduled  (only)  as  the  result  of  failure  (or  heavy 
congestion).  The  results  presented  here  suggest  that  this 
method  could  provide  superior  performance  at  reduced  cost. 
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CHAPTER  5 


(SOURCE  AND)  DESTINATION  BUFFERING  CONSIDERATIONS 


FOR  STREAM  TRAFFIC  COMMUNICATION 


5.1  lQl£S^U££i2Q 

In  communicating  stream  information  via  a packet 
switched  network  rather  than  the  traditional  circuit 
switched  (or  dedicated)  network  there  arise  some  unique 
problems  which  require  solution.  Among  these  problems  is 
the  packaging  of  information  into  packets.  Some  ad  hoc 
solutions  to  this  problem  are  discussed  in  section  5.2.  But 
the  chief  concern  of  this  chapter  is  dealing  with  the  vari- 
able delay  imposed  by  a packet  switched  network.  Several 
methods  of  limiting  this  variability  in  the  output  by  desti- 
nation buffering  are  considered.  3y  delaying  the  output  of 
the  first  message  of  a stream^  one  may  limit  the  frequency 
and  duration  of  gaps  in  the  output  (i.e.^  intervals  of  time 
in  which  no  data  is  available  to  output).  Such  gaps  are 
undesirable  since/  to  some  extent/  they  destroy  the  rhythm 
of  the  output  and  thus  hinder  the  intelligibility  of  the 
information.  Clearly/  as  the  first  message  delay  is  in- 
creased/ the  frequency  and  duration  of  gaps  decreases.  In 
the  limiting  case  of  infinite  destination  buffering  delay  it 
is  guaranteed  that  zero  gaps  will  occur  in  the  output  but 
this  of  course  destroys  the  interactive  nature  of  the 


MS 


communi cat  Ion 


Therein  lies  the  tradeoff  which  is  examined 


in  this  chapter/  namely  qaps  versus  delay. 

By  assuming  statistical  i nd*‘penden ce  of  the  delay  ex- 
perienced by  stream  packets  traversing  the  network/  we  may 
analyze  this  tradeoff.  The  assumption  is  somewhat  justi- 
fied/ but  fails  to  be  true  under  certain  conditions/  which 
we  partially  examine.  Based  on  this  independence  assumption 
we  can  derive  some  general  analytic  results.  Numerical 
results  are  presented  for  the  exponential  distribution  in 
some  detail  and  to  a lesser  extent  for  a class  of  r-stage 
Erlangian  distribution  of  delay.  Following  the  theoretical 
performance  results/  we  present  some  results  of  a simulation 
which  allows  for  the  relaxation  of  some  of  the  assumptions 
of  the  model.  This  is  accomplished  with  the  use  of  a trace 
driven  simulation  of  the  various  buffering  methods. 

5.2  isadioa  ilra^ggy 

Whenever  stream  information  is  produced  by  the  source 
in  units  which  are  smaller  than  a full  packet  (or  message) 
we  have  the  option  of  sending  the  information  in  partially 
full  packets.  Doing  so  reduces  the  time  required  to  create 
a packet/  but  increases  the  throughput  requirement  of  the 
source  since  packets  contain  a non-zero  amount  of  overhead. 
Once  again/  one  finds  a tradeoff  between  throughput  and 
delay ! 

This  tradeoff  is  pictured  in  Figure  5.1  for  the  fixed 
rate  ARPANET  LPC  algorithm  (see  CCohe  76]).  There  is  a 
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linear  increase  in  packet  fill  time  as  we  move  from  the 
minimum  of  one  parcel  to  the  maximum  of  14  parcels  per  pack- 
et. (A  parcel  consisting  of  67  bits  contains  one  set  of  LPC 
coefficients  and  as  such  is  the  smallest  unit  which  may  be 
separately  interpreted  at  the  destination.)  The  slope  is 
19.2  msec/parcel.  It  is  clear  that  an  inverse  relationship 
holds  for  the  throughput  in  terms  of  either  packets/sec  or 
bits/sec  as  shown.  If  the  network  protocol  uses  a full 
packet  time  to  send  a partially  full  packet  (e.g./  a slotted 
channel  as  in  the  ARPA  SATNET  CKlei  733  and  PRNET 
[Kahn  753)^  then  the  bits/sec  curve  is  of  no  use. 

Two  extreme  approaches  exist  for  a sending  strategy: 
(1)  minimize  delay  (at  whatever  the  resulting  throughput 
requirement);  and  (2)  minimize  the  throughput  requirement 
(at  whatever  the  resulting  delay).  It  is  clear  that  the 
choice  of  sending  full  packets  satisfies  (2).  However^  it 
is  not  so  clear  that  sending  the  smallest  packets  minimizes 
the  overall  delay. 

Let  f(p)#  x<p)^  c<p)/  w(p)/  and  D(p)  be  defined  as  fol- 
lows for  the  stream  packets  given  that  packets  of  size  p 
parcels  are  in  use. 

f(p)  = the  fill  time» 

x(p)  = the  network  transmission  time/ 

c(p)  = the  network  propagation  and  processino  time/ 

w(p)  = the  average  network  waiting  time/  and 

0(p)  = the  average  destination  buffering  time. 
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packets  when  w is  likely  to  be  small  enough  to  guarantee 
that  overall  delay  will  not  exceed  a tolerable  level. 

An  examination  of  this  approach  has  shown  that  on  a 
microscopic  scale  (i.e./^  packet  by  packet)  it  is  infeasible 
since  it  was  found  that  future  delay  is  not  well  predicted 
by  past  delay  in  general.  The  correlation  results  presented 
in  section  5. 3. 3. 2 attest  to  this.  Therefore  durino  the 
short  term  it  is  wise  to  use  fixed  p. 

On  a macroscopic  scale  some  adjustment  of  p may  prove 
useful.  This  may  be  accomplished  by  using  the  destination 
monitoring  techniques  discussed  in  section  5.3.  As  network 
delay  increases  one  may  wish  to  use  smaller  p.  This  may  be 
done  only  when  w is  not  significantly  increased  by  this 
decrease  in  p.  If  network  delay  increases  quite  substan- 
tially then  an  increase  in  p is  in  order  (to  reduce  w). 

We  have  offered  here  only  ad  hoc  suggestions  for  the 
sending  process.  With  a good  model  for  w(d)  some  useful 
analytic  results  would  be  attainable.  8ut  since^  as  pointed 
out  earlier^  the  throughput  requirement  curves  are  fairly 
flat  past  p=5/  the  payoff  gained  by  such  analysis  is  likely 
to  be  minimal.  We  shall  therefore  direct  our  attention  to 
the  receiving  process. 

5.3  Adacilyg  rg£giviQg 

The  aim  of  adaptive  receiving  of  stream  traffic  is  to 
output  the  information  with  as  close  to  the  same  timing  with 
which  it  was  originally  generated  under  the  constraint  of 
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allowing  interactive  cominuni  cation  (i.e.^  reasonably  smalt 
delay).  It  is  particularly  important  in  the  case  of  speech 
to  reduce  the  frequency  and  duration  of  gaps  in  order  to 
insure  acceptable  intelligibility.  In  this  section  we  first 
describe  three  methods  of  gap  control  using  buffering^  and 
two  playback  strategies  for  handling  those  gaps  which  slip 
through  the  control  procedures.  The  performance  of  the 
various  schemes  is  then  analyzed.  Me  then  present  some 
numerical  results  for  the  theoretical  performance^  followed 
by  simulation  results  comparing  the  various  procedures. 

5.3.1  Gse  caotccl 

Since  variable  delays  do  occur  in  a packet  switched 
networks  if  no  smoothing  were  done/  gaos  would  occur  in  the 
output  of  a packet  stream.  That  is/  there  would  be  periods 
of  time  in  which  no  data  was  available  for  playout.  This 
has  disastrous  effects  on  the  underst andabi I ity  of  speech  in 
particular  and  in  general  violates  the  definition  of  stream 
traffic.  Therefore  gap  prevention  or  reduction  is  neces- 
sary. 

One  method  of  gap  control  is  to  delay  the  output  of  the 
beginning  of  a sentenre  (i.e./  a period  of  activity  by  the 
sender)  by  an  amount  D/  which  may  be  selected  at  the  begin- 
ning of  each  sentence  based  on  sampling  of  previous  delays. 
Another  scheme  is  to  slow  the  playout  in  the  absence  of  the 
next  packet  and  speed  playout  when  an  excess  of  waiting 
packets  exists.  (The  effect  of  this  on  intelligibility  may 
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not  warrant  its  use.)  One  may  mix  these  two  to  arrive  at 
still  others  or  invent  new  ones.  We  shall  consider  only  the 
first  scheme. 

The  first  packet  of  a sentence  will  be  delayed  by  an 
amount  called  the  destination  wait  time  denoted  as  D.  The 
choice  of  D should  be  made  large  enough  to  reduce  the  fre- 
quency of  gaps  to  a tolerable  level  but  small  enough  to 
retain  the  interactive  nature  of  a conversation.  In  a later 
section  we  shall  consider  several  adaptive  methods  for 
choosing  D. 

5.3.2  Plsy^ifk  l££|iQd£ 

Unless  D is  very  larger  there  is  a non-zero  probability 
that  a gap  will  occur.  The  purpose  of  this  section  is  to 
describe  two  methods  of  dealing  with  this  eventuality.  The 
first  method  (method  E)  would  expand  the  playout  time  of  a 
sentence  in  order  to  include  all  packets  in  the  output  pro- 
cess. The  second  method  (method  I)  preserves  the  timing  at 
the  expense  of  ignoring  some  late  arriving  packets  (or  par- 
tial packets).  These  two  approaches  lie  at  opposite  ends  of 
a continuum  of  choices  for  dealing  with  gaps.  At  constant 
delay  (e.q./  dedicated  channel)  the  two  extremes  are 
equivalent.  Also  if  D is  infinite  the  extremes  coincide. 
However#  with  finite  0 and  variable  delay  the  extremes 
separate.  The  separation  increases  as  delay  variability 
increases  or  as  0 decreases.  At  infinite  delay  variability 
method  E requires  infinite  time  to  output  a sentence  (of 
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more  than  one  packet)#  while  method  I would  output  only  the 
first  packet  of  a sentence  and  ignore  any  other  packets. 
Thus  method  E preserves  the  information  at  the  expense  of 
the  interpacket  timing#  and  method  1 preserves  the  inter- 
packet timing  at  the  expense  of  discarding  some  information. 
With  finite  delay  variability  and  finite  D one  can  envision 
methods  which  lie  between  the  two  extremes.  For  example# 
one  may  wish  to  discard  only  those  packets  which  arrive  both 
late  and  out  of  order  (i.e.#  a packet  is  discarded  if  its 
successor  is  currently  being  output).  Such  a scheme  has 
properties  of  both  methods  E and  I.  The  time  axis  is  ex- 
panded when  a packet  is  just  late#  yet  some  data  may  be  dis- 
carded in  order  to  preserve  "reasonable"  timing.  Another 
example  is  to  limit  the  expansion  of  time  and/or  the  frac- 
tion of  discarded  data  to  a certain  amount  and  switch 
methods  if  the  threshold  is  exceeded. 

An  important  consideration#  which  shall  not  he  dis- 
cussed here#  is  the  filling  of  gaps  (i.e.#  with  silence  or 
something  else).  Several  alternatives  (which  are  beyond  the 
scope  of  this  discussion)  have  been  used  and  are  discussed 
in  CForg  761. 
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5. 3. 2.1  EiCaDdSd  lime  a^ls  (Wethad  £) 

In  this  method  content  is  regarded  as  the  most  impor- 
tant part  of  the  stream  of  packets.  Therefore#  it  is 
desired  to  deliver  all  packets  which  are  produced  by  the 
source  of  the  stream  traffic.  Timing  is  considered 
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important  only  to  the  extent  that  a packet  is  delivered  in 
the  same  sequence  in  which  it  was  generated.  A "late"  pack- 
et is  one  which  arrives  after  it  is  desired  for  output 
(i.e.#  after  its  predecessor  has  finished  playout).  In  this 
method  a late  packet  causes  all  succeeding  packets  to  be 
delayed  in  playout^  thus  expanding  the  time  axis  or  the 
length  (in  time)  of  the  sentence.  An  example  of  this  is 
illustrated  in  Figure  5.2(a). 

This  is  the  general  approach  taken  by  the  network 
speech  compression  research  group  at  the  University  of 
Southern  California  Information  Sciences  Institute  (ISI). 
In  the  ISI  scheme  extremely  late  out  of  order  packets  are 
i gnored. 


5. 3. 3. 2 Uai£  data  ISDSiLe^  ^^gthgd  I) 

In  method  I timing  is  considered  to  be  of  primary  im- 
portance. A packet  (or  partial  packet)  will  be  used  only 
when  it  is  not  late.  By  discarding  late  information  it  is 
always  possible  to  retain  the  sentence  length  (in  time). 
Figure  5.2(b)  shows  the  previous  example  sentence  from  Fig- 
ure 5.2(a)  passing  through  method  I.  Notice  that  sentence 
time  is  preserved  but  that  more  gaps  occur  than  in  method  f. 
This  is  true  in  general  as  we  shall  see  in  later  sections. 
Method  I is  in  use  by  experimenters  at  the  Massachusetts 
Institute  of  Technology  Lincoln  Laboratory. 
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5.3.3  P£ClfiC!SaQ£fi 

An  examination  of  the  performance  of  several  adaptive 
receiving  techniques  is  presented  in  this  section.  Ue  begin 
by  considering  some  assumptions  which  render  the  system 
tractable  for  analysis.  The  analysis  is  then  performed.  An 
attempt  to  characterize  communication  quality  is  made. 
Numerical  results  based  on  the  analysis  are  presented  in 
detail  for  the  exponential  d i st ri but i on/  and  in  lesser  de- 
tail for  a class  of  Frlangian  distributions  TKlei  753. 
Simulation  is  used  to  show  some  results  with  the  assumptions 
relaxed. 

Our  model  of  the  system  is  pictured  in  Figure  5.3.  A 
period  of  activity/  called  a sentence/  is  initiated  by  the 
sender  at  some  time.  (This  corresponds  to  the  detection  of 
no-silence  in  speech  for  example.)  As  time  progresses  the 
sentence  is  broken  up  into  a sequence  of  segments  called 
packets  or  messages.  A finite  amount  of  time  f(i)  is  re- 
quired to  fill  packet  i.  Also  associated  with  each  packet  i 
is  a network  transit  delay  y(i).  Let  t(i)  = f(i)  ♦ y(i)  be 
the  source-to-dest ination  delay  of  packet  i.  We  make  the 
following  assumptions  regarding  the  random  variable  t(i): 
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AiSU&l2iiQD£  q1  tbe  !SQSl£i 

1 . Independence 

We  shall  assume  that  each  t(i)  is  chosen  indepen- 
dently from  a probability  distribution  function 
Slz)  = PrCt (i) <zD . 

2.  Stationarity 

We  shall  further  assume  that  Slz)  is  stationary 
(i.e./  *:(z)  is  not  a function  of  time). 

While  the  assumption  of  stationarity  is  used  in  order 
to  obtain  analytic  results^’  the  algorithms  (as  we  shall  see) 
adapt  to  nonstationary  behavior. 

Followino  the  source-to-dest i nat i on  delay  there  is  a 
(possibly  zero)  destination  buffering  delay  for  each  packet. 
In  particular^  this  buffering  delay  takes  the  value  D for 
the  first  packet  of  a sentence.  As  we  see  later#  0 is  a 
function  of  S(z)  and  the  particular  delay  monitoring  algo- 
rithm in  use. 

It  is  assumed  that  each  packet  requires  f(i)  to  empty 
(as  well  as  to  fill).  If  a packet  arrives  after  its  prede- 
cessor has  completed  emptying#  then  a gap  occurs  in  the  out- 
put stream  (indicated  in  the  figure  by  crosshatching  at  the 
output).  Obviously  there  is  a tradeoff  between  the  value  D 
and  the  frequency  with  which  gaps  occur.  We  shall  therefore 
use  these  two  parameters  as  our  measures  of  performance. 
Let  us  first  consider  the  validity  and  implication  of  the 
major  assumptions  of  the  model. 
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3. 3. 3.1  of  QglUSCiL  dsiil 

The  purpose  of  this  section  is  to  present  some  observed 
distributions  of  network  delays  and  provide  guidelines  for 
producing  tractable  approximations  of  these  di st ri but  ions. 
Both  ARPANET  measurement  and  simulation  were  used  obtain 
delay  distribution  examples. 

i 

The  approximation  derived  from  the  independence  assump- 
tion of  Kleinrock  CKlei  64D  would  suggest  an  Erlangian 
(i.p.^  the  sum  of  exponentials)  distribution  of  delay.  This 
assumption  renders  the  model  of  each  channel  an  M/P/1  queue 
which  yields  an  exponentially  distributed  system  time.  The 
I time  through  a series  of  queues  therefore  would  follow  an 

Erlangian  distribution. 

Exact  formulas  have  been  derived  for  the  distribution 
j of  message  delay  in  an  isolated  non-interf ered  path  in  a 

networks  for  Poisson  arrivals  and  deterministic  service 
time^  by  Rubin  (see  CRubi  753).  Our  problem  is  to  find  the 
delay  distribution  of  a stream  traffic  source  in  a general 
network  which  does  not  seem  to  fit  the  assumptions  of  the 
previous  two  models.  Namely  the  stream  packets  must  not  be 
allowed  to  change  length  as  they  proceed  through  the  net- 
work/ and  our  problem  would  be  trivial  in  a non-int erf ered 
path . 


5. 3. 3. 1.1  BeaSUESffiSQt 

The  measurement  experiments  described  in  Chapter  4 pro- 
vide the  distributions  presented  here.  In  order  to 
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eliminate  the  start-up  effects  the  first  100  samples  have 
been  discaraed  in  each  case.  The  resulting  histograms  are 
plotted  in  Figure  5.4(a-d).  In  (a)  and  (b)  there  is  a tri- 
fflodal  behavior  exhibited.  This  shows  the  effect  of  the 
periodic  routing  update  procedure  quite  clearly  (again). 
The  first  mode  corresponds  to  the  round-trip  delay  experi- 
enced by  a message  which  always  finds  empty  queues  along  its 
path.  The  third  mode^  falling  about  45  msec  after  the  first 
mode#  corresponds  to  the  delay  of  those  messages  and 
corresponding  end-to-end  acknowledgements  (RFNM)  which  must 
wait  for  a total  of  45  msec  on  queues  while  traversing  the 
network.  This  amount  of  time  corresponds  to  the  transmis- 
sion time  of  two  routing  update  packets#  the  processing  of 
four  such  packets  (approximately  two  in  the  slower  316 
nodes)#  or  some  combination  of  the  above.  The  second  mode# 
occurring  roughly  10  msec  beyond  the  first#  is  probably  due 
to  the  packet  (or  its  RFNM)  waiting  behind  the  processing  of 
one  routing  update  along  the  way.  Significance  could  prob- 
ably be  assigned  to  the  other  peaks  as  well  but  this  would 
belabor  the  point. 

As  we  move  to  five  hops  (c)  and  then  to  ten  hops  (d)  we 
notice  that  the  trimodal  behavior  ceases  to  appear  and  the 
histograms  take  the  shape  of  an  Erlangian  density.  Perhaps 
Kleinrock's  independence  assumption  produces  an  acceptable 
approximation  for  stream  traffic  as  well! 


5. 3. 3. 1.2  SiffiUifiiifiD 

The  results  of  the  measurement  are  biased  by  the  large 
overhead  associated  with  the  routing  update  procedure.  We 
therefore  resort  to  simulation  in  order  to  remove  this  ef- 
fect/ to  control  the  level  of  interfering  traffic  in  the 
network/  and  to  study  one  way  delay  instead  of  round-trip 
delay.  The  simulation  had  the  following  characteristics: 

Topology:  A ring  of  21  nodes  and  21  full  duplex  chan- 

nels 

Channel  capacity:  5C000  bits/sec 

Traffic  pattern:  A uniform  traffic  matrix  of  exponen- 
tially distributed  message  lengths  with  a mean  of 
500  bits  and  Poisson  arrivals  as  background  traff- 
ic. Inserted  with  this  was  one  stream  traffic 
source  sending  to  a destination  ten  hops  away  at  a 
rate  of  one  500  bit  packet  every  250  msec. 

The  background  traffic  was  set  to  three  particular  lev- 
els in  order  to  produce  .1/  .5  and  .9  channel  utilization. 
One-way  network  delay  was  measured  for  the  stream  traffic 
and  the  resulting  histograms  appear  in  Figure  5.4(e-g).  The 
.1  load  histogram  <e)  suggests  a shifted  exponential  density 
with  perhaps  an  impulse  at  the  shift  value.  For  load  .5  and 
.9  (f  and  g respectively)  the  histograms  have  more  of  an 
Erlanoian  shape  (with  an  impulse  in  the  .5  case). 
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Figure  5.4(b).  Delay  histogram  (measurement,  2 hops). 
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Figura  5.4(f).  Dalay  hiitogram  (limulation,  load*  .5). 
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Figure  6.4(gK  Delay  hiitogram  (titnulation,  load*  .9). 


5. 3. 3. 1.3  l£a£tsbi£  access Is dllSQS 
In  order  to  obtain  some  analytic  results  we  must  assume 
that  the  delay  distribution  may  be  characterized  in  a 
mathematically  tractable  form.  In  this  section  we  consider 
some  tractable  approximations  to  the  measured  and  simulated 
delay  distributions.  We  have  already  suggested  two  - (1) 

the  shifted  exponential  and  (2)  the  Erlangian  class. 

ihi  shiiisd  &sa&Q£Qllai  dlsiributisa 
This  distribution  may  be  formulated  as 


set)  = 


11  -(t-b)/w 


where  b is  the  amount  of  sh:ft  and  b't^w  is  the  mean  value. 
Figure  5.5  shows  the  histogram  of  the  simulation  in  Figure 
5.4(e)  plotted  together  with  a shifted  exponential  with  the 
same  mean  value  and  with  b equal  to  the  minimum  observed 
value.  This  appears  to  be  a fairly  close  fit. 
lti£  friaca  qI  di^tcibuiiaas 

The  shifted  exponential  was  a close  fit  to  one  of  the 

sample  distributions.  We  now  consider  the  Erlang  family 

/ 

which  may  be  formulated  as 


dS(t)  = 


-:fr) 


-rt/x 


(r-1) ! 


J 


The  formula  has  been  written  so  that  mean  value  is  indepen- 


dent of  r.  As  Kleinrock  CKlei  75D  points  out  on  page  124/ 
the  standard  deviation  of  this  distribution  is  x/sqrt(r). 
(Figure  5.26  shows  this  density  function  for  several  values 
of  r . ) 

By  selecting  r one  may  change  the  coefficient  of  varia- 
tion (defined  to  be  the  standard  deviation  divided  by  the 
mean)  between  zero  and  one.  All  the  observed  distributions 
have  coefficients  of  variation  in  this  range.  Figure  5.6 
shows  those  observed  histograms  (which  appear  to  be  Erlangi- 
an)  plotted  together  with  the  member  of  the  Erlang  family  of 
the  appropriate  r and  mean  value.  This  appears  to  be  a 
close  fit/  particularly  for  the  simulation. 
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Figura  S.6(b). 


Erlang  fit  to  dalay  hiitogram  (maaturamant,  10  hope). 
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Figura  5.6(c).  Erlang  fit  to  dalay  histogram  (simulation,  load  ■ .5). 
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where^  as  before^  b is  the  amount  of  the  shift  and  w = x - 

b. 

Figure  5.7  shows  the  fit  of  the  histogram  of  the  ap- 
propriate shifted  Erlangian  distributions  together  with  the 
observed  histograms.  Notice  the  close  fit  for  all  but  the 
measured  distributions  for  one  and  two  hops.  The  two  which 
are  not  well  approximated  with  the  shifted  Erlang  class 
could  it  appears  be  approximated  by  a waited  sum  of  shifted 
exponentials/  but  we  shall  not  attempt  this  here. 
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Figure  5.7(b).  Shifted  Erlang  fit  to  delay  hittogram  (maaiuremant,  2 hops). 
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Figura  5.7(d).  Shifted  Erlang  fit  to  dalay  hiatogi 
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5. 3. 3. 2 iOiiSDfiQdsQSS  q1  osit^sck  {2aLa^ 

In  the  study  of  stochastic  processes^  statistical  in- 
dependence is  often  assumed  to  make  analysis  simpler  (even 
possible).  We  make  use  of  such  an  assumption  here.  In  this 
section  an  heuristic  argument  is  presented  to  justify  this 
assumption.  This  argument  is  then  partially  substantiated 
by  measurement  and  simulation. 

We  assume  that  one  of  the  characteristics  of  a stream 
source  is  that  packets  are  emitted  at  relatively  large  in- 
tervals compared  to  the  interval  during  which  a packet  occu- 
pies a single  channel  in  the  network.  more  precisely  we 
have  the  property  that  the  average  interarrival  time  of 
packets  from  the  stream  source  t >>  § the  average  time  spent 
waiting  for  and  using  a channel.  This  is  both  an  assumption 
of  (a)  moderately  low  throughput  for  the  stream  source^  and 
(b)  low  overall  traffic.  If  (a)  were  not  true^  then  the 
source  becomes  classified  as  a high  throughput  source.  If 
(b)  were  not  the  case  then  the  network  delays  may  exceed 
those  desired  for  interactive  stream  communication. 

With  this  assumption  we  see  that  each  packet  enters  the 
network  and  is  likely  to  be  far  along  its  way  to  the  desti- 
nation before  its  successor  enters  the  network.  Therefore 
each  packet  arrives  to  find  the  network  in  a state  which  is 
"independent”  of  the  influence  of  his  predecessors  which  are 
no  longer  in  sight.  Each  packet  then  receives  an  indepen- 
dent "look"/  if  you  will/  at  the  network.  We  conclude  that 
the  delay  experienced  by  successive  packets  should  be 
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approximately  independent. 

Let  us  examine  the  measurement  and  simulation  results. 
A necessary^  byi  QQt  condition  for  statistical 
independence  is  that  of  linear  independence.  Linear  depen- 
dence may  be  tested  simply  by  computing  the  correlation 
coefficient  CFell  57]  <p.  221)  of  a sequence  of  delays  ui th 
itself  shifted  by  an  amount  j.  By  changing  the  shift  j we 
obtain  a sequence  of  correlation  coefficients.  This  is 
related  to  the  "autocorrelation"  sequence  defined  in 
COppe  753  (p.  384).  Our  sequence  however  is  normalized  by 
first  subtracting  the  mean  from  each  value  (which  gives  the 
"autocovar iance"  sequence)  and  dividing  the  result  by  the 
variance  of  the  original  sequence.  The  result  of  this  com- 
putation for  j = 2#  .../  200  is  shown  in  Figure  5.F(a-d) 
for  the  measured  sequences  of  delay  and  in  Figure  5.8(e-g) 
for  the  delay  sequences  produced  by  the  simulation  described 
in  the  previous  section.  The  figures  show  that  a near  zero 
linear  dependence  exists  for  the  measured  delay  of  5 hops  or 
more  and  for  a load  of  .1  in  the  simulation.  For  a load  of 
.5  successive  delays  are  less  than  30  percent  correlated  but 
beyond  j=1  there  is  little  correlation.  As  expected  delay 
at  a load  of  .9  is  highly  correlated  for  the  first  few 
values  of  j.  Parts  (a)  and  (b)  of  Figure  5.8  show  a regular 
pattern  of  correlation.  This  is  further  evidence  of  the 
periodic  nature  of  the  interference  of  the  routing  update 
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For  comparison  to  Figure  5.8  we  include  Figure  5.9 
which  shows  the  correlation  coefficient  computed  in  the  same 
way  for  a sequence  of  pseudo-random  numbers.  Its  appearance 
is  much  like  Figure  5.8(c-f). 

With  the  argument  partially  suoported  by  measurement 
and  simulation^  we  conclude  that  the  assumption  of  statisti- 
cal independence  is  not  completely  unreasonable.  We  shall 
proceed  with  this  assumption  as  at  least  the  first  approxi- 
mation to  the  behavior  of  the  actual  system. 

5. 3. 3. 3 Gag  Qisbabliili: 

Having  stated  and  considered  some  of  the  implications 
of  the  assumptions  of  the  model/  we  are  now  prepared  to 
begin  the  analysis  of  that  model.  We  begin  by  finding  ex- 
pressions for  the  probability  of  gap  occurrence  as  a func- 
tion of  the  delay  distribution  and  R(D)  the  distribution 
which  characterizes  the  choice  of  the  destination  wait  value 
0.  Let  us  begin  with  method  E. 

5. 3. 3. 3.1  Gac  BCSba^liity  fsc  Dia^^aBil  0!£thQd  I 

Under  the  assumption  that  fill  and  network  delays  are 
each  independent/  we  may  find  the  probability  with  which 
gaps  occur  in  the  output  process.  Let  y(i)  be  the  network 
delay  and  f(i)  be  the  fill  time  of  message  i.  Let  t(i)  = 
y(i)  + f(i)  - y(i-1)  - f(i-1)  be  the  difference  in  delay 
experienced  by  message  i and  i-1.  Then  the  ft(i)>  are  mutu- 
ally independent  and  identically  distributed  random 
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variables  since  we  have  assumed  that  the  ■Cy(i)>  and  'Cf(i)> 
are.  Let  the  nth  partial  sum  of  tt(i)>  be  P(n)  = P(n-1)  + 
t(n)#  and  let  P(0)  = 0.  Notice  that  P(n>  = y(n)  + f(n>  - 
Cy(0)  +f(0)].  Now  let  us  define  the  strict  ascending  ladder 
indices  <N((c)>  for  <P(n)>  as  in  CPrab  653  (p.  140)  as  fol- 
lows: 

N(0) =0 

N(1)  = minCn  : P<n)>G) 

N(k)  = min<n  ; n>N<k-1)  and  P(n) >P (N ( k-1 ) ) ) 

To  proceeds  we  need  the  following  theorem. 

IhfiSIfiffi-  A gap  in  the  output  process  in  method  E oc- 
curs at  message  i if  and  only  if  i is  a ladder  index  N(k) 

for  some  k>0  and  P(i)>D/  where  0 is  the  destination  wait 

time. 

££22ii  An  output  gap  occurs  if  and  only  if  a message 
arrives  after  it  is  needed  for  playout.  If  message  zero 
begins  filling  at  time  zero/  then  the  time  at  which  message 
i arrives  at  the  destination  is 

f(0)  + f(1)  + ...  + f(i)  + y{i) 

(i.e./  the  time  to  fill  all  previous  messages  plus  the  time 
to  fill  message  i plus  the  network  delay  of  message  i). 

Suppose  that  P(i)  < D for  all  i.  Then  for  each  i we 

have 

P(i)  = y(i)  + f(i)  - (y(0)  +f(0))  < 0 for  all  i.  (5.?) 

The  time  that  message  one  is  required  for  playout  is 

f(0)  + y(0)  + 0 + f<0) 
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(i.e.#  the  time  of  arrival  of  message  zero  plus  the  destina- 
tion wait  time  plus  the  time  to  playout  message  zero).  From 
equation  5.2  we  have 

yd)  + f(1)  + f(0)  < y<0)  + f(0)  ♦ 0 + f<0). 

Therefore  message  one  is  not  late  and  does  not  cause  a gap. 
Equation  5.2  may  be  rewritten  as 

f(i)  + y(i)  < y(0)  + f(0)  + D. 

By  adding  f(0)  + f(1)  + ...  * f(i-1)  to  both  sides  we  have 
i-1  i-1 

5]  f(j)  + f(i)  + y(i)  < X}  + y<0)  + f(0)  + D. 

i=0  j=0 

This  states  that  message  i arrives  at  or  before  its  required 
time.  Hence  no  gaps  will  occur  when  P(i)  < D for  all  i. 

Suppose^  on  the  other  hand/  that  there  exists  an  i such 
that  P(i)  > D.  Choose  k1  to  be  the  minimum  such  i.  Then  k1 
is  a ladder  index  by  definition/  since  it  is  the  first  oc- 
currence of  a value  of  P(i)  greater  than  D.  We  have 
f(0)  + y(0)  + 0 < f(k1)  + y(k1) 
and  therefore 

k1-1  k1-1 

f<0)  + y<0)  + 0+53  * f(k1)  + y(k1) 

j=0  j=0 

and  hence  the  first  gap  occurs  at  ladder  index  k1  whose 
value  P(k1)  is  first  to  exceed  0. 

After  the  occurrence  of  this  gap  at  k1/  message  k1  is 
output  immediately  upon  arrival  and  therefore  message  k1+1 
will  be  needed  at  time 
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k1 

2] f <j)  + y(k1)  + f (k1) 

3=0 

The  next  gap^  if  any/  will  occur  at  message  k?  where  k2  is 

the  minimum  i for  which 

i k1  i-1 

2f(j)  + yCi)  > + Z ^<3) 

j=0  j=0  i=ki 

(i.e./  message  i arrives  after  it  is  needed  for  playout). 
The  following  three  statements  are  each  equivalent  to  the 
last . 

f(i)  + y(i)  > f(k1)  + y(k1) 

f(i)  + y(i)  - f(0)  - yCO)  > f(k1)  ♦ y(k1)  - y(0)  - f(0) 
P(i)  > P(k1) 

Therefore/  since  k2  was  chosen  as  the  minimum  such  i/  k2  is 
the  ladder  index  immediately  following  k1 . 

We  may  now  prove  the  general  case  by  induction.  Assume 
that  the  first  n-1  gaps  occurred  at  the  first  n-1  ladder 
indices  whose  values  are  greater  than  D.  Let  I be  the  last 
ladder  index  at  which  a gap  occurred.  After  the  occurrence 
of  this  gap  at  1/  message  I is  output  immediately  upon  ar- 
rival (as  was  message  k1)  and  therefore  message  l->^1  will  be 
needed  at  time 
I 

2]  f (j)  ♦ yd)  ♦ fd) 
i=0 

The  next  gap/  if  any/  will  occur  at  message  11  where  11  is 
the  minimum  i for  which 
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1 I i-1 

2]f(j)  + y(i)  >2^f(j)  ♦ y(l)  ♦ E^^j) 

3=0  i=0  3=1 

(i.e./  message  i arrives  after  it  is  needed  for  playout). 
Once  again/  the  following  three  statements  are  each 
equivalent  to  the  last. 

f(i)  + y(i)  > f(l)  + y(l) 

f(i)  + y<i)  - f(0)  - y(0>  > f(l)  + yd)  - y<0)  - f(0) 
P(i)  > P(l) 

Hence  11  is  the  ladder  index  immediately  following  1. 
Therefore  each  ladder  index  whose  corresponding  value  is 
greater  than  D has  an  associated  gap#  and  a gap  occurs  only 
at  a ladder  index  whose  corresponding  value  is  greater  than 

D . 

Q.E.D. 

Thus  the  probability  that  the  kth  message  in  a sentence 
produces  a gap  is  equal  to  the  probability  that  k i s a 
ladder  index  and  P(k)  > 0#  i.e.# 

PrCgap  | k/03  = PrCk  is  a ladder  index  and  P(k)>D] 
According  to  CPrab  65]  (p.  1A1)  we  have 

PrCgap  j k/DD  = PrCP(k)>0/  P (k-1 ) >0/ . . ./  P{1)>0/  and 

P(k)>DJ 

Since  P(k)>D  implies  P(k)>0  and  P(1)  is  guaranteed  to  be 
greater  than  zero#  we  have 

PrCgap  | k/0]  = PrCP(k)>D/  P (k-1 )>0/ . . . / P(2)>0] 

Assume  that  the  (fill  plus  network)  delay  is  distributed 
with  probability  distribution  function  S<y)/  that  the  delay 
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of  the  first  message  in  the  sentence  is  t/  and  that  D is  the 


destination  wait  time.  Then 

k-2 

PrCgap  1 k^t^DD  = Cl -S (t  + 0) 3 C 1-S ( t )] 


We  may  remove  the  condition  on  k to  obtain  the  probability 
that  a gap  occurs  at  a randomly  selected  message  in  a sen- 
tence of  length  n#  given  t and  0/  i.e.^ 

1 n k-2 

PrCgap  | t,D/n3  =-  C1-S(t+D)  3 C1-S  (t  >3 

n k=2 


Removing  the  condition  on  t we  have 

1 n-2  faa  k 

PrCgap  | D/n3  = - 2Z  / Cl -S< t +0)3 C 1 -S ( t ) 3 dS(t) 
n k = 0 Jt  = 0 


Removing  the  condition  on  0 we  have 

1 n-2  ran  /“oo  k 

PrCgap  /n3  = - X)  / / Cl -S( t+D)3 Cl -S ( t ) 3 dS(t)dR(D) 

n k=0  •'0  = 0 •'t=0 

(5.3) 


where  R(0)  is  the  distribution  function  of 
wait  time  0.  Since 
n 


n-1 

Ex 

i=0 


t-x 

1-x 


the  destination 


we  have 


n-1 

1 /•oo  ran  1-C1-S(t)3 

PrCgap  /n3  = - / / C1-S(t+0)3 dS(t)dR(0) 

r\J{>=0  Jt  = D S(t) 

(5. A) 
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We  shall  evaluate  this  expression  nunerically  in  succeeding 
sect  ions. 

5. 3. 3. 3. 2 Sac  BEsfeabiiiJix  fac  Eiaxlzisk  lathasi  I 
since  there  is  no  expansion  of  the  time  axis  in  method 
we  expect  that  when  a message  exceeds  the  delay  of  the 
first  message  by  more  than  D a gap  results.  The  destination 
arrival  time  of  message  k is  (as  before)  given  by 
k 

E + y<k) 

j=0 

The  time  at  which  message  k is  required  is 

k-1 

f<0)  + yCO)  + D + 

, ) = o 

Hence  we  have  a gap  whenever 


X)f(j)  + yik)  > f(0)  + y(0>  + 0 + y)f(j) 

1=0  i=0 

or 

f(k)  + y(k)  > f(0)  + y(0)  + 0. 

Therefore  the  probability  of  a gap  at  message  k^  given  that 
f(0)  + y(0)  = t and  D is  the  destination  wait  time^  is  1- 

S(t+D).  For  sentences  of  length  n messages  we  have 

1 n-1  n-1 

PrCgap  I t^D^n]  = - 2 C1*S(t*D)D  = C1-S(t+D)3 

n k = 1 n 


Removing  the  condition  on  t we  have 


PrCgap  | O^nD  = j C1-S(t+D)3  dS(t) 

n Jt  = 0 

Removing  the  condition  on  f>  we  have 

n-1  ^00  /•oo 

PrCgap  | nD  = / / Cl-S(t+D)D  dS(t)dP(D>  (5.5) 

n Jo^D  Jt-Q 

Comparing  the  right  hand  side  of  this  to  that  of  equation 
5.3  we  notice/  as  expected/  that  this  expression  is  larger. 
Therefore/  in  general/  the  gap  probability  is  lower  in 
method  E than  in  method  I.  Numerical  results  appear  below. 

5. 3. 3. 4 Daiai:  BCfidisiers 

In  the  previous  two  sections  we  have  assumed  a distri- 
bution for  the  destination  wait  time  D.  The  purpose  of  this 
section  is  to  explore  some  possible  delay  predictors  each 
yielding  a distribution  function  for  D.  Three  schemes  are 
discussed  and  all  are  based  on  monitoring  which  may  be  per- 
formed at  the  destination  node.  The  first  scheme  predicts 
delay  variation  by  computing  the  range  of  delay  for  previous 
samples.  We  have  found  that  this  scheme  "learns"  quickly 
but  performs  poorly  when  occasional  long  delays  occur.  In 
order  to  ignore  these  "spikes"  in  delay/  and  thus  achieve 
smaller  D/  we  introduce  the  second  scheme  which  views  only 
that  portion  of  the  range  below  a threshold.  The  third 
scheme  attempts  to  track  the  changing  delay  by  modifying  an 
internal  counter  to  predict  message  arrival  times. 

The  basic  idea  then/  in  each  scheme/  is  to  measure  the 


169 


delay/  and  based  on  the  informat'fon  gained/  to  make  some 
intelligent  choice  for  h at  each  sentence  boundary.  It  is 
clear  that  the  larger  is  D the  fewer  are  the  gaps  which  will 
occur  in  the  output  stream.  However/  large  D tends  to  des- 
troy the  interactive  nature  of  the  communication.  Ideally  a 
scheme  would  "optimally"  balance  these  two  properties.  It 
is  difficult  to  define  optimality  here.  However/  the  fol- 
lowing general  statement  holds:  Large  destination  wait  time 
and/or  frequent  gap  occurrence  each  yield  a poor  quality  of 
communication. 

5. 3. 3. 4.1  m-jaaBlS  £3096 

In  both  playback  methods/  if  the  delay  of  a message 
within  a sentence  exceeds  the  delay  of  the  first  message  of 
that  sentence  by  more  than  D/  then  at  least  one  gap  will 
occur.  The  total  range  of  the  previous  m (m>1)  samples  is  a 
pessimistic  estimate  for  the  difference  between  maximum 
delay  within  a sentence  and  the  delay  of  the  first  message 
in  that  sentence.  It  is  pessimistic  because/  on  the  aver- 
age/ the  first  delay  will  fall  somewhere  between  the  ex- 
tremes. The  problem  is  to  find  the  distribution  of  the 
range  of  m samples.  Our  derivation  below  follows  that  of 
CGumb  67D  <pp.  97-98).  We  begin  with  the  joint  probability 
that  t is  the  minimum  (denoted  t=min)  and  t+z  is  the  maximum 
(denoted  t+2=max)  among  m samples  each  drawn  independently 
from  distribution  S(y). 


m 


PrCt=inin^  t + z=max  j m samples] 

= PrCat  least  one  sample  = t#^ 

at  least  one  sample  = t't-z^  and 

m-2  samples  fall  in  the  interval  Ct^t+z3  3 


m-2 

= m dS(t)  (m-1)  dS(t  + z)  CS  (t+z)-S(t) 3 


By  removing  the  conditions  on  t and  t+z  we  find  that  the 
distribution  of  range  R(D  | m)  = PrLrange<0  1 m samples3  is 
given  by 


R(D  I m) 


m-2 

(m-1)CS(t+z)-S Ct)3  dS(t+z) 


dS(t> 


Yielding 


R(D  I ffl) 


CS(t  + D)-S(t>  3 


m-1 


dS(t) 


(5.6) 


This  equation  cannot  be  reduced  further  without  knowledge  of 
S(t)  . 


5. 3. 2. 4. 2 ffi-aaffioifi  Bartial  caoss 
The  full  range  estimate  functions  fairly  well  (as  we 
show  later)  but  it  is  not  without  fault.  Since  the  entire 
range  is  used/  isolated  cases  of  high  delay  cause  an  unduly 
large  value  to  be  chosen  for  the  next  destination  wait  time. 
For  this  reason  we  wish  to  examine  the  partial  range  of  m 
samples  where  the  k highest  values  are  ignored  (m>1  and 
k<m-1).  We  wish  to  find/  as  before/  the  distribution  R(0  | 
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m^k)  of  the  partial  range  given  the  distribution  Sly)  from 
which  the  m samples  are  chosen. 

we  begins  as  before/  with  the  joint  probability  that  t 
and  t+z  are  among  the  m samples  drawn  from  S(y)/  where  t is 
the  minimum/  and  t+z  is  the  maximum  among  the  m-k  smallest 
s amp  les . 

PrCdrawing  such  a sampled 

= PrCat  least  one  sample  = t/ 

at  least  one  sample  = t+z/ 

in-k-2  samples  are  in  Ct/t+z3/  and  there  are 
k samples  which  exceed  t+z  D 


m-k-2  /m-? 


= mdS<t)(m-*1)dS<t+z)CS(t  + z>-S(t)D 


(T)"- 


S(t+z)3 


Therefore 


R(0  I m/k)  = m(m-1) 


CD 


rr 

Jt  = 0 Jz  = 0 


m-k-2  k 

CS(t+z)“S(t)]  C1-S<t+z)3  dS(t+z)dS(t) 

(5.7) 


5. 3. 3. 4. 3 A d£ia)(  tEitllSC 

A third  method  of  delay  prediction  was  developed  by 
James  Forgie  CForg  76a]  of  the  Massachusetts  Institute  of 
Technology  Lincoln  Laboratory/  Lexington/  Massachusetts.  It 
compares  the  time  stamp  of  an  arriving  message  with  the 
"arrival  clock"  and  adjusts  that  clock  by  plus  (or  minus) 


one  count  if  the  message  arrived  earlier  (or  later)  than 
predicted  by  the  clock  value.  At  the  beginning  of  a sen- 
tence a quantity  called  the  "reconstitution  delay"  c is  sub- 
tracted from  the  arrival  clock  to  set  the  playout  clock.  It 
is  important  to  note  that  the  value  of  c is  chosen  at  the 
beginning  of  a conversation  by  the  user.  The  waiting  packet 
begins  playout  when  the  playout  clock  reaches  the  value  of 
the  time  stamp  within  the  packet.  Time  is  measured  in  units 
equal  to  a frame  size  (19.2  msec  for  Linear  Predictive  Cod- 
ing/ LPC/  algorithm  used  experimentally  for  speech  communi- 
cation in  the  ARPANET/  see  CCohe  76D)  and  the  clock  is  in- 
creased by  one  at  each  frame  time. 

Figure  5.10  illustrates  the  operation  of  the  algorithm. 
The  first  message  of  the  sample/  over  which  the  arrival 
clock  alx)  is  adjusted/  arrives  at  the  destination  and  ini- 
tiates the  clock  value  (assumed  to  be  at  time  zero).  The 
time  stamp  in  each  message  is  shown  as  the  ordinate  of  a dot 
at  the  point  (x/y)/  where  x is  the  arrival  time  of  the  mes- 
sage at  the  destination  and  y is  the  time  stamp  within  that 
message.  Notice  that  an  "early"  arrival  has  its  time  stamp 
above  the  arrival  clock  and  a "late"  arrival  appears  below. 
The  arrival  clock  therefore  attempts  to  "track"  the  points 
(x/y)  by  adding  one  at  points  where  y > a(x)  and  subtracting 
one  where  y < a(x).  Were  the  clock  unadjusted/  the  value 
would  be  floor(x)  (the  largest  integer  less  than  or  equal  to 
x).  Let  the  clock  difference  l(x)  be  defined  to  be  alx) 
floor(x).  Then  alx)  predicts  the  time  stamp  of  a message 
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arriving  at  time  x with  time  stamp  y should  (have)  arrive(d) 
at  X - (a(x>  -y);  or  more  precisely  f loor ( x-a(x) -y) . Since 
a(x)  and  y are  integers  we  have  that  the  predicted  arrival 
of  a message  arriving  at  time  x with  time  stamp  y is  y - 
l(x).  At  the  arrival  of  message  m (the  start  of  a sentence) 
the  playout  clock  p(xm)  is  set  to  a(xm)-c#  and  then  steps  at 
frame  boundaries.  Playout  begins  when  p(x)  > ym  or 
equivalently  r units  after  the  predicted  arrival  time  of 
message  m (i.e./  at  time  ym  - Kxm)  ♦ r).  Hence  for  the 
value  D as  previously  defined  we  have  D = ym  - Kxm)  + c 
-xm.  We  shall  disallow  negative  D and  thus  we  write 

D = maxCO^  c + ym  - xm  -l(xm)>.  (5.8) 

This  algorithm  may  be  modeled  by  a Markov  chain 
CKlei  75]  where  the  states  represent  the  clock  difference 
value  l(x)  and  transitions  occur  at  message  arrival  points. 
When  a message  arrives  with  delay  less  (more)  than  predicted 
by  the  current  state  k/'  then  a transition  to  state  k + 1 (k-1) 
occurs.  As  the  function  l(x)  starts  at  value  zero  the  Mar- 
kov chain  will  start  in  state  zero.  Suppose  t is  the  delay 
of  the  first  message  (i.e./  that  message  which  initiates 
arrival  clock).  Given  that  the  current  state  is  the  pro- 
bability of  a transition  from  state  k to  k+1  is  equal  to  the 
probability  of  experiencing  a delay  less  than  t + kf^  where 
f is  the  frame  size.  Likewise^  the  probability  of  a transi- 
tion from  k to  k-1  is  equal  to  the  probability  of  a delay 


reater  than  or  equal  to  t - (k-1)f.  Otherwise#  we  remain 
in  state  k if  the  delay  lies  between  t - kf  and  t - (k-Df. 
Figure  5.11  displays  this  graphically  for  the  continuous 
distribution  function  S(t). 


S(t(k-1)f)  S(t-kf) 


- S(Mcf> 

Figure  5.11.  Delay  tracker  Merkov  chain. 

/ 

These  transition  probabilities  insure  the  existence  of 
an  irreducible  (sub-)  Markov  chain.  If  the  random  variable 
is  bounded  then  the  chain  will  have  a finite  number  of 
states.  Since  the  chain  is  irreducible  we  may  equate  the 
flow  into  and  out  of  state  k+1  as  follows: 

flow  in  = flow  out 

p(k)CS(t-kf)3  + p(k+2)Cl-S(t-(k+1)f)D 

= p(k+1)C1-S(t-kf )3  * p(k+1)CS(t-(k+1)f)3 
where  pCk)  is  the  steady-state  probability  of  being  in  state 
k (we  have  assumed  that  S(t)  is  continuous). 

Solving  for  p(k)  we  have 

p(k  + 1)C1-S(t-kf)  + S(t-(k*1)f  )3  - p(k*2)C1-S(t-(k  + 1)f  )3 


Equation  (5.9)  defines  a recursive  procedure  for  calcu- 
lating the  steady-state  state  probabilities  for  the  f'arkov 
chain.  Unfortunately#  in  the  current  form  it  requires  an 
infinite  number  of  steps  in  general.  One  can#  however# 
achieve  an  approximation  using  the  following  general  proper- 
ties. There  exists  some  I for  which  S(t-(l+1)f)  = 0 and 
S(t-lf)  > 0 (i.e.#  the  random  variable#  delay#  is  bounded 
below).  For  a random  variable  which  has  a finite  lower 
bound  and  finite  mean  value  we  have  that#  for  any  e > 0 we 

may  find  a finite  positive  integer  I'  such  that  1 - S(t+l'f) 
< e.  Therefore  the  steady-state  of  being  in  any  state  num- 
bered -I '-1  or  less  is  less  than  e.  If  L=l+l*+1#  we  need 
only  perform  L steps  (plus  the  usual  normalization  step)  to 
achieve  a value  for  the  state  probabilities  each  within  e/L 
of  the  correct  value. 

Thus  far  we  have  considered  only  the  steady-state  pro- 
babilities. One  may  also  wish  to  consider  the  m-step  state 
probabilities  given  that  we  begin  in  state  zero.  Let  p(k 
I m#0)  be  the  probability  of  being  in  state  k after  m transi- 
tions starting  from  state  zero.  Then 
p(0  l0#0)  = 1 

p<k  I l#0)  = p(k-1  I l-1#0)CS(t-(k-1) f)3 

♦ p(k  I l-1#0)CS<t-(k-1)f)-S(t-kf )3 

♦ p(k+1  I 1-1 #0)C1-S(t-kf)) 

This  defines  another  recursive  procedure  which  terminates 
after  m iterations  for  at  most  2m+1  states. 

by  computing  the  state  probabilities  as  indicated  we 
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find  the  distribution  function  R(0).  Note  that  the  proba- 
bility that  L(x)  = k is  just  p(k).  Assumina  that  a message 
arrives  to  start  a sentence  at  time  * with  time  stamp  y then 
fro*  equation  5.8  we  have 

R(D  I x^y^c)  = PrCmaxCO^c+y-x-l (x)>  < D 3 

= PrC  c+y-x-l(x)  < D 3 

= PrC  l<x)  > c+y-x-O  3 

1+1 

RtO  I x^y/C>  = ^ I p(i) 

i =f loor ( c+y-x-D) 


The  quantity  x-y  is  the  message  delay  (both  fill  and  net- 
work) which  we  have  earlier  assumed  to  be  distributed  as 
S(t).  Therefore 


R(D  I c) 


J/*oo  I*  1 

" E 

t=0  i=f 


p<i)  dS<t) 


f loor (c-t-0) 


We  s^all  not  evaluate  this  numerically/  but  later  show  some 
simulation  results  in  the  use  of  this  algorithm. 

We  have  assumed  a constant  c in  the  analysis  thus  far 
and  this  was  the  case  in  the  actual  implementation  of  the 
algorithm  at  Lincoln  Laboratory  (i.e./  constant  for  the 
length  of  a conversation).  An  automatic  method  of  adjusting 
c to  the  network  dynamics  has  been  developed  (again  by  James 
Forgie).  After  an  initial  value  is  given  to  c/  it  is  in- 
creased if  a gap  occurs  and  decreased  if  a string  of  s 
(currently  contemolated  to  be  100)  messages  pass  through  the 


j 
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system  without  gaps-  This  may  again  be  modeled  by  a Karkov 
chain. 

The  new  chain  will  be  three  dimensional  with  states 
indexed  by  k the  old  state  labels/  i the  current  length  of 
the  gapless  string/  and  the  current  value  of  c.  Let 
G(k/i/c}  = PrCgap  occurs  given  l(x)=k/  gapless  string  length 
= i/  and  c = the  reconstitution  delayD.  Transitions  occur 
at  message  arrivals  as  before.  It  is  simpler  to  visualize 
transitions  as  taking  two  steps.  First  consider  the  part  of 
a transition  which  is  governed  by  the  gap  probabilities. 
The  possible  transitions  and  their  probabilities  are  as  fol- 
lows 

PrC(k/i/0>  to  (k/i+1/0)3  = 1 - G{k/i/0) 

PrC(k/S/c)  to  (k/0/C-1>D  = 1 - G(k/S/c) 

PrC(k/i/c)  to  (k/i+1/C)D  = 1 - G(k/i/c) 

PrC(k/i/c>  to  (k/O/c+DD  = G(k/i/c) 

Next  consider  the  part  of  a transition  due  to  a change  in 
delay. 

PrC(k/i/c)  to  <k/i/c)]  = S(t-(k-1)f)  - S(t-kf) 
PrC(k/i/c)  to  <k+1/i/c)3  = S(t-kf) 

PrC(k/i/c)  to  <k-1/i/c)3  = 1 - S(t-(k-1)f) 

We  attempt  no  further  solution  to  this  problem/  but  rather/ 
we  later  present  simulation  results. 


179 


5. 3. 3. 5 C2|iQ!UQl£fi£iQQ 

before  proceeding^  we  need  a basis  for  comparing  the 


"quality"  of  communication  in  our  stream  traffic  context. 
Among  the  parameters  affecting  the  quality  of  communication 
in  the  system  under  discussion  are  the  probability  of  an 
output  gap  occurrence  and  f+t+Dx  the  speaker-to- I i st ener 
(end-to-end)  delay.  The  end-to-end  delay  and  the  gap  proba- 
bility are  both  influenced  by  the  destination  wait  time  D. 
Ideally  one  would  choose  to  operate  at  the  point  0=0  and 
G=0.  Unfortunately  this  cannot  be  achieved  in  general. 
There  is  a tradeoff  between  these  two  parameters.  Experi- 
ence has  shown  that  we  can  tolerate  nonzero  G up  to  some 
threshold  value  CForg  76D  beyond  which  the  quality  degrades 
rapidly.  For  the  experiment  reported  in  CForg  763  roughly  6 
percent  of  the  packets  caused  gaps  and  the  quality  was  con- 
sidered to  be  acceptable.  It  is  likely  that  a similar 
behavior  is  true  for  D (i.e./  beyond  some  value  of  f+t+D  the 
communication  ceases  to  be  of  an  interactive  nature).  Stu- 
dies regarding  the  effect  of  transmission  delay  on  telephone 
conversations  reported  in  CKrau  673  and  CKlem  673  show  that 
the  threshold  is  above  .6  sec  and  below  1.8  sec. 

We  wish  to  choose  a quality  function  Q(t)^G)  which  has 
this  threshold  behavior.  One  such  family  of  functions  is 

1/(2n-1) 


So«e  members  of  this  family  are  shown  in  Figure  5.12. 

Suppose  we  define 

Q(D^O>  = q(D/d>  and 

Q(0^6)  = q(6/g)  for  some  given  thresholds  d and  g. 

It  remains  to  fill  in  the  DG  plane  so  as  to  continuously 
coincide  with  the  axis  functions.  As  mentioned  earlier/  the 
point  of  highest  quality  is  the  origin.  Then  quality  should 
decrease  as  we  increase  the  distance  from  the  origin.  One 
reasonable  choice  for  U(D/G)  is  then 


(5.11) 


Another  choice  would  be  the  product  of  the  axis  qualities 

i .e./ 

Q(D/G>  = Q(D/0)Q(0/G). 

Any  quality  function  will  define  contours  of  constant 
quality.  We  shall  define  the  region  of  acceptability  to  be 
the  region  enclosed  between  the  positive  D and  G axes  and 
the  DG  plane  projection  of  the  contour  whose  value  is  Ci(d/0) 
a Q(0/g).  This  region  is  independent  of  n and  an  example  is 
shown  in  Figure  5.13.  Other  contours/  however  increase  in 
distance  from  the  origin  with  increasing  n.  This  property 
is  illustrated  in  Figures  5.14  and  5.15.  Therefore  by  ad- 
justing n we  may  change  the  importance  of  the  origin  as  the 
optimal  operating  point.  If/  for  example/  one  wishes  to 
assign  the  same  value  of  quality  to  all  points  in  the  region 
of  acceptability  then  an  infinite  value  of  n should  be 


Figure  5.1 2. 


Gap  protMbility  . probability 


chosen.  With  cofflinunication  quality  defined  by  equations 
5.10  and  5.11  we  may  proceed. 

5. 3. 3. 6 NUSSrlSSi  £Slui£S 

The  purpose  of  this  section  is  to  present  some  numeri- 
cal results  by  substituting  a delay  distribution  into  the 
formulas  derived  in  the  preceding  sections.  Exact  results 
will  be  given  for  an  exponential  distribution  of  delay.  For 
the  more  general  class  of  r-stage  Erlang  distributions  nu- 
merical integration  is  used. 

5. 3. 3. 6.1  Iti£  isegnsQ^iil  distd^UllCQ  of 

For  the  purpose  of  demonstration#  we  choose  the  ex- 
ponential distribution 

-t/x 

S(t)  = 1 - e 

Admittedly  this  is  a poor  choice  for  the  distribution  of 
network  delay!  The  measurements  presented  earlier  substan- 
tiate that.  This  distribution#  however#  has  a large  coeffi- 
cient of  variation  when  compared  to  the  measured  distribu- 
tions. It  therefore  provides  a lower  bound  on  performance# 
which  should  hold  for  less  tractable  but  more  realistic  dis- 
tributions. Some  r-staqe  Erlang  distributions  (more  what 
one  expects  from  a network)  are  treated  in  the  next  section. 

Since  R(0)#  the  distribution  of  destination  wait  time# 
is  required  in  the  gap  probability  formulas#  we  shall  first 
find  R(D  | ffl#k).  We  begin  with  equation  5.6.  Substituting 


"1 


r 


for  S(t)  we  have 


n-1 


R(D  I IP) 


roof  -t/x  -(t  + 0)/xl  1 

= m/L*  "*  J “® 

^ = 0 X 


-t/x 


dt 


m-1 


r -D/x1  /•oo  1 
= n,[l-e  J / 

•'t=0  X 


-mt  / X 

e dt 


R(D  I m) 


ir-1 


(5.12) 


The  mean  value  of  D given  m is  given  by 


m- 


D(m)  = 


I 

1 /•  f -O/xl 

- / D[l-e  J 


m-2 

-0/x  j -D/x 
e I e dD 


m-1  m-2  i /m-2\  /•  oo  -<i+1)0/K 

De  dD 


m-1  m-d  ^ /m-d\  r ao 

= — E<-1)  ( )/  t 

X 1=0  \i/«'D  = n 


m-1  m-2  i /m-2\  x 

(,)m 


X i=G 


(i+1)  (i+1) 


m-J  i-1  /m-1\  1 

E <-’>  ( ) - 

i=1  \ i / i 


= X 


m-2  i-1  /m-2\  1 m-1  i-1  /m-2\  1 

{J- 


X m-1 

B(m-1)  + 1-(1-2) 

m-1 


2 = 1 
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Therefore^ 


TO  - 1 1 
D(m>  = X 2)  * 

i = 1 \ 


(5.13) 


Kext  we  evaluate  R(D  | m#k).  Beginning  with  equation 
5.7  and  substituting  for  S(t)  we  have 

m-2 


R(0  I B^k)  = b(b-1) 


f f t 

Jt  = C .72=0 


B-k-2 

-t/x  -(t*2)/x1  -k(t+2)/X 

J 


e -e 


1 -(t+2)/x  1 -t/x 

- e - e dz  dt 

X X 


ni-2  1 r "Bt/x 


= in(»-1) 


1/ 

X A=0 


e dt 

m-k-2 


1 -(k+1)z/x 

- e dz 

X 


to-2\  m-k-2  i /m-k-2\  /•D  -(i+k  + 1)z/x 


(to-2\  m-k-2  1 /m-k-2\  /•D 

. ) ,?o  . )L 


dz 


-(i  + k + DD/x 


/m-2\  m-k-2  i /m-k-2\  1-e 
R(D  I mxk)  = x(m-1)(  j ^ C-1)  ^ j 


i + k + 1 


(5.14) 


R(D  I 


m-1 


which  agrees  with  equation  5.12. 

It  is  useful  to  check  to  see  that  R(0  | m#k)  has  the 
proper  characteristics  of  a probability  distribution  func- 
tion. Clearly  R(U  | m^k)  = 0 (as  desired).  Let  us  shew 
that  the  limit  of  R(0  | m#k)  as  D approaches  infinity  is 
one . 

rm-2\  in-k-2  i /m-k-2\  1 


(m-2\  in-k-2  i /m-k-2\ 

) (-1)(  )- 
k / i=0  \ i / i 


i+k  + 1 


(m-2\  m-k-Z  i /m-k-2\ 

) L ( ) — 

k / i=0  \ i / (i 


(i+1 ) (i+2) .. .(i+k) 

♦1 )(i+2)... (i+k) (i*k*1) 


1 m-1  i-k-1  /m-1 

— E 

k!  i=k+1 


(7)”- 


1 ) (i-2) ...(i-k) 
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J 


D{m+1#k)  - D(mrk1  = x 2^  (-1) 

i=k  + 1 

m-1 

- X S (-1) 

1=k  + 1 


iji . ; ; 

i-k-1  /"’"K /■’“K  ^ 

\ i A k / i 


i = k + 1 


10  ■(•r)i 


in-1  i-k«1  r/ni\  /i“1\  1 

= X Y. 


i~m-1  /m-1\  1 


♦ (-1) 


k / m 


X m 


£ (-1) 

If  1 =k  ♦ 1 


7-k-1  /mWi-l’ 

W\  •«  / 


X JE_ 


— E (-1) 

ink  ! i =k+  1 


i-k-1  /ni\  (i-1)  ! 


Qi  1 -1 ; : 

(i-1-k) 


k m 

X k d 1-(1-2) 

(-1)  

ink ! k z 

dz 


z = 1 


or  £(ir  + 1#k)  = O(m^k)  ♦ x/m 


(5.16) 


This  partially  defines  a recursive  procedure.  What  remains 
is  to  find  the  starting  point  for  each  k (i.e.^  0(k+2#k)  ). 
From  equation  5.15  we  have 


D(k*?/k)  = X 


k+1 

T. 


(5.17) 


Beginning  with  equation  5.17  and  performing  repeated  appli- 
cation of  5.16  we  have 

0(k*3/k)  = £(k+2/k)  + x/(k+2) 


D (m#  k) 


m-1  1 

X X 

i=k+1  i 


From  equation  5.13  it  follows  that 

D(m^k)  = !i(m/0)  - 0(k+1/0)  (5.18) 

Figure  5.16  shows  the  behavior  of  ^(m/k)  for  m from  2 
to  100  and  several  k.  The  k=0  curve  is  logarithmic  in 
shape/^  having  its  most  rapid  ascent  at  the  beginning  and 
then  tapering  off.  This  is  a useful  property  in  the  selec- 
tion of  both  m and  k.  The  difference  between  Dlm/O)  and 
£(m-1^0)  (i.e.^  B=1/(ni-1)  ) is  the  curve  from  which  the 
&(a/k)  curves  emanate.  This  shows  that  the  difference 
between  successive  values  of  D(m/k)  is  very  small  beyond  say 
m=20 . 

Let  GE(n#m/k)  be  the  gap  probability  given  that  (1) 
each  sentence  contains  n messages^  (2)  D is  the  partial 
range  of  m samples  with  the  k largest  samples  discarded^  and 
(3)  the  delay  distribution  is  exponential.  Initially  we 
shall  drop  the  m and  k parameters.  Then  equation  5.3  yields 
the  following 
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I n - 2 ^00  f-  aa 

GE(n)  = - E / / 

n 1=0  yo=0  Jt  = 0 


- e dt  dR(D) 

X 


1 n-2  /•(DO  “0/x  r a 

= - E / e / 

n 1=0  •'0=0  J \.-l 


n~2  /"(DO  “0/x  ^(»1  -t(i  + 2)/x 

e / - e dt  dR(D) 

=0  X 


1 n-2  /•«  -0/x  1 

= - E / e — 

n i=0  •'0=0  1 + 2 


dR(D) 


(5.19) 


Suppose  k=0.  Then 


ni-2 


1 n-2  1 r 00  -O/xf  -D/xl 

6E(n/in^0)  = — ■'  / e 1— e I 
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m-l  -0/x 

e do 

X 
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1 n-2  1 
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(5.20) 


This  function  decreases  as  m or  n increase  (for  n>2) 


Figure  5.17  shows  a family  of  m versus  GE  for  n 


2/ 


4, 


fi. 


16^  24^  32.  Each  curve  falls  raoidly  initially  and  then 

flattens  out  as  m increases. 

For  general  k we  substitute  the  right  hand  side  of 

equation  5.14  into  5.19  to  obtain 

1 n-2  1 ^00  -0/x  n-2 

GECn^m/k)  = - ^ — / e (m-l) 
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53  ( - e dD  (5.21) 
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6E(n<.m,k)  = 5Z  = <k  + 1)  GE  (n,m,0)  (5.22) 

mn  i=0  i+2 


Figure  5.18  shows  GE  versus  m for  n=1 6 and  several  k 


we  define  Gl(n^m^k)^  the  gap  probability  for  method 
simildPly  to  GECn^n^k).  Beqinning  with  equation  5.5  we  have 
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(5.23) 


Suppose  k=0.  Then 
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Unlike  GE#  GI  increases  with  n.  It  has  a finite  upper 
bound  of  1/2in  as  sentences  grow  very  large.  Figure  5.19 
illustrates  a family  of  61  curves.  For  general  equations 
5.23  and  5.14  yield 
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From  the  derivation  following  5.21  leading  to  5.22  we  have 

(k+1)(n-1) 

GlCn^m^k)  — — s (k+1)GI(n/m^0)  (5.25) 

2mn 


Figure  5.20  shows  GI  versus  m for  various  k with  n=16. 
&Q  tb£  £1  kl£2£  !!! 

GE(n/ffl^0)  and  GKn/m/^O)  each  have  limiting  values  of 
zero  as  m approaches  infinity.  O(m^O)  is  unbounded  however. 
One  must  allow  k to  grow  large  in  order  to  bound  0.  Suppose 
we  let  k=m-j  (j>1).  Then 
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This  approach  puts  £ at  zero  but  forces  GE  and  GI  outside 


the  acceptable  region.  Suppose  we  let  m and  If  approach 
infinity  such  that  k/m  approaches  the  value  a I0<a<1).  Then 


k+1  n-2 
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a n-2 
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limit  D(ffl^k)  = limit  ^ 
m— ► m— * i=k  + 1 i 


An  expression  for  this  last  limit  must  be  found  in  oroer  to 
analytically  determine  the  optimum  value  of  a (based  on  the 
quality  function  defined  earlier).  We  begin  by  recognizing 
the  following  inequality 

m-1  1 


m 1 /•  m-1  1 

E - < / - dt  < E 

i=k  i yt=k-1  t i=k-1  i 


Now  notice  that 


^ m 1 

Jf  - Qt  = log(m)-log(k)  = log(m/k) 

t=k  t 


Taking  the  limit  as  m approaches  infinity  we  see  that 

m 1 11  m-1  1 m-1  1 

limit  E “ ~ limit  - - — ♦ E " * limit  E 

m-*oo  i=k  i m-*oo  m k-1  i=k-1  i m-»  i=k-1  i 


Hence  it  must  be  that 

ffl-1  1 m-1 

limit  E " ~ limit  log  — = *log(1/a)  = -loq(a) 

m-*oo  i=k  + 1 i m-*oo  k + 1 
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We  wish  to  minimize  the  normalized  distance  from  the  origin; 
i .e./ 

2 2-11/2 

•)  • C) 

The  square  root  may  be  dropped  without  affecting  the  minimi- 
zation. Notice  that  the  limit  of  GE  and  61  may  be  written 
as  a (different)  constant  (actually  a function  of  n)  times 
the  parameter  a.  Therefore/  the  form  of  the  function  to  be 
minimized  is 


( 


2 2 
-loQ(a)\  /aK' 


Setting  the  derivative  (with  respect  to  a)  equal  to  zero  we 
have 

2 2 
2log(a)/1\  /K\ 

— W * ” 


2 /"  \ 

a = -(  — ]log(a) 

VdK/ 


(5.26) 


Checking  the  value  of  the  second  derivative  at  the  value  of 
a thus  obtained  we  have 

2 2 2 2 


2 2 

-2(- ) log(a)  + 2(-)  ( - ) +2 

\a/  \a/  \d/ 


0 


Using  the  value  in  equation  5.26  we  obtain 
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2 - 


loq(a) 


> 0 


for  0<a<1 


Therefore  the  solution  is  indeed  a minimum.  Numerical  solu- 
tion of  5.26  (for  n=16)  yields  a=.0801  for  method  E and 
a=.0546  for  method  I.  This  gives  corresponding  optimum 
points  of  6E=.0119  at  0=2. 52x  and  GI=.0256  at  0=2. 91x. 
Since  one  would  be  required  to  remember  the  k+1  largest 
values  (and  the  minimum  value)/  the  process  which  allows  m 
and  k to  grow  arbitrarily  large  is  computationally  infeasi- 
ble. The  knowledge  of  our  proximity  to  the  optimum  is  how- 
ever quite  important. 

In  Figure  5.21  we  plot  GE  versus  0 for  k = iJ  and  various 
n.  The  choice  of  m is  quite  critical.  For  the  case  of  n=32 
it  can  be  shown  that  the  acceptable  range  for  m is  approxi- 
mately 16  to  24.  If  m is  smaller  GE  is  too  large;  if  m is 
larger  then  0 is  too  large.  (The  larger  region  of  accepta- 
bility for  GI  causes  a somewhat  wider  choice  of  acceptable 
m.  ) 

In  order  to  examine  the  behavior  of  different  values  of 
k/  we  fix  n (at  16)  and  plot  GE  versus  D in  Figure  5.22. 
Notice  that  performance  increases  (i.e./  normalized  distance 
from  the  origin  decreases)  as  k increases  (with  correspond- 
ing increase  in  m).  That  is/  for  those  curves  plotted/  for 
any  m/k  pair  there  exists  m*  such  that  the  performance  at 
m*/k+1  exceeds  the  performance  at  m/k.  Also/  for  each  m 
there  is  an  optimum  value  of  k (which  maximizes  the  quality 
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function).  Fiqure  5.23  shows  the  value  of  the  optimal  k for 
each  m<200.  The  average  slope  of  the  functions  approximates 
the  value  of  the  parameter  a.  Fiaure  5.24  gives  the  perfoi — 
mance  for  in<200  at  the  optimum  k for  each  m.  Also  shown  is 
the  limiting  performance.  The  performance  approaches  the 
limit  quickly  at  first  and  therefore  one  may  achieve  perfor- 
mance quite  close  to  the  optimum  for  small  m (and  therefore 
small  k).  To  see  this  more  clearly  we  have  plotted  in  Fig- 
ure 5.25  the  fraction  by  which  the  performance  at  each 
m^k-optitnal  point  exceeds  the  limiting  performance.  That 
is#  if  L(Gi)  is  the  limiting  performance#  then  CQ(m#k-opt)- 
L(0)!]/L(Q)  is  the  ratio  plotted  versus  m in  the  figure. 
Notice  that  convergence  to  the  limit  is  faster  than  a nega- 
tive exponential  (which  would  appear  as  a straight  line  in 
this  figure). 

rule  of  thumb  for  choosing  m and  k is  as  follows. 
Choose  m>4D  and  k approximately  " percent  of  m for  method  F 
and  5.5  percent  for  method  I.  We  see  in  Figure  5.25  that 
using  such  a rule  for  the  exponential  distribution  of  delay# 
n=16  and  the  given  regions  of  acceptability  achieves  perfor- 
mance within  5 percent  of  the  limiting  performance.  In  sub- 
sequent sections  we  relax  these  assumptions  and  find  that 
the  rule  continues  to  provide  good  performance. 
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Figure  5.23.  Optimal  k for  given  m. 
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Figura  5.24.  Pirformanca  at  m^-optimal  (Mathod  E). 
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5. 3. 3. 6. 2 ih£  ^billed  s&QOQSQtiai  dlurlbuli&D 
In  section  5. 3. 3. 1.3  «e  noted  that  the  shifted  exponen- 
tial (and  the  shifted  Erlangian)  could  sometimes  be  used  to 
approximate  an  observed  delay  distribution.  Analytic 
results  for  this  distribution  are  shown  in  this  section  to 
be  similar  to  the  results  for  the  exponential  distribution. 

The  key  observation  is  that  equations  5.6^  5.7/  5.4  and 
5.5  are  invariant  under  a linear  shift  (translation  of 
axes) . For  example 


R(D  1 tn) 


00 

CS(t+0)-S(t) 3 dS(t) 

0 


■r 

yt-b=o 


m-1 


CS (t-b+D)-S (t-b)] 


dS(t-b) 


The  resulting  formulas  are  therefore  identical  to  those 
for  the  exponential  distribution  with  one  exception.  The 
mean  value  x must  be  replaced  with  the  parameter  w = x-b. 
This  appears  in  the  formula  for  D(m/0)  (i.e./  equation 
5.13).  Let  b^K'k/b)  be  the  mean  range  of  m samples  with  the 
highest  k discarded/  given  the  delay  is  distributed  with 
exponential  distribution  shifted  by  b.  Then 
x-b 

0(m/k/b)  = D(m/k/0)  (5.27) 

X 

Therefore  as  b increases  0(m/k/b)  decreases.  As  t ap- 
proaches X the  limiting  range  is  zero/  which  is  expected 
since  the  limiting  distribution  is  that  of  a constant  valued 
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random  variable.  The  results  of  the  previous  section  there- 
fore apply  here  after  an  appropriate  scaling  of  the  D values 
by  the  factor  <x-b)/x. 


5. 3. 3. 6. 3 Itie  E.Li.aQaiaQ  Idisiiz  si 

Section  5. 3. 3. 1.3  shows  that  the  Frlangian  (and  shifted 
Erlangian)  family  of  distributions  provide  useful  approxima- 
tions to  observed  distributions.  In  this  section  results 
for  a family  of  Erlangian  distributions  appear.  The  family 
to  be  considered  appears  in  Figure  5.26  as  a family  of  pro- 
bability density  functions.  Symbolic  integration  of  equa- 
tions 5.6#  5.7#  5.4  and  5.5  becomes  exceedingly  complicated 
for  an  Erlangian  distribution.  Therefore  we  resort  to  a 
numerical  integration  technique  in  order  to  solve  for  D#  GF 
and  Gl. 

Using  the  so  called  parabolic  rule  of  numerical  in- 
tegration Child  743  (p.  96)#  we  evaluate  equations  5.7#  5.4# 
and  5.5.  The  results  appear  in  Figure  5.27  as  a set  of 
points  in  the  0-GE  plane.  We  have  fixed  n at  16  and  show 
the  values  obtained  for  r-5#  10#  and  50#  with  m=2#  10#  20# 
30#  40#  50#  and  100#  and  k=0#  1#  2#  3#  4#  5#  and  10  (and 
k<m-1 ) . 

There  many  similarities  between  these  results  and  those 
for  the  exponential  distribution  (i.e.#  r=1 ) . Connecting 
the  points  of  constant  k we  see  that  the  hyperbolic  shape 
persists.  We  also  notice  that  the  distance  between  constant 
k curves  shrinks  rapidly  with  increasing  k#  as  before. 


i 

i 


1 

= 

s 


i 


Notice  that  6E  (and  Gif  not  shown)  decreases  much  more  ra- 
pidly^ as  £ increases/  than  for  the  r=1  case.  (In  fact  a 
different  scale  was  required  to  show  some  degree  of  detail.) 
This  is  not  unexpected;  as  mentioned  earlier  the  r=1  case 
was  expected  to  provide  a lower  bound  on  performance.  It  is 
expected  that  as  variability  of  delay  decreases  less  buffer- 
ing will  be  necessary  to  provide  the  same  degree  of  gap  pro- 
bability. This  is  indeed  the  case  and  at  the  limit  as  r 
approaches  infinity  delay  becomes  constant  and  buffering  is 
no  longer  required  to  reduce  the  gap  probability. 

Relative  distance  from  the  origin  (i.e./  communication 
quality)  is  based  on  the  region  of  acceptability  which  we 
have  not  specified  for  this  family  of  distributions.  We 
have  circled  the  points  m = 40/  50/  and  100  with  k = 3/  4/ 
and  10  respectively  (i.e./  those  closest  to  the  rule  of 
thumb).  Notice  that  points  provide  a rather  good  balance  of 
mean  delay  with  gap  probability. 
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Figura  5.27.  Ptrformanc*  with  Erlang  distributions.  (r~5,10,50) 
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5.3. 3. 7 SiouiatlQQ  ££SUil& 

Some  assumptions  were  made  in  earlier  sections  in  order 


to  provide  a tractable  model  of  the  system.  Fven  with  these 
assumptions^  analytic  results  were  obtained  only  for  a small 
class  of  distributions.  We  therefore  have  resorted  to  simu'- 
lation  in  order  to  relax  some  of  the  assumptions  made. 
Presented  in  this  section  are  the  results  of  that  simula- 
tion. 

The  simulation  program  is  driven  by  a sequence  of  de- 
lays. Therefore  it  is  possible  to  remove  the  assumption  of 
statistical  i ndependenre  among  samples.  This  assumption  is 
relaxed  by  using  actual  delay  strings  from  ARPANFT  measure- 
ments and  from  simulation. 

It  is  also  possible  to  remove  the  assumption  of  fixed 
sentence  size.  Suggested  by  the  early  work  of  Norwine  and 
Murphy  CNorw  383^  we  have  used  a geometric  distribution  of 
sentence  length  <in  terms  of  packets)^  with  a mean  which 
corresponds  roughly  to  the  4.14  seconds  reported  in 
CNorw  383  as  the  average  length  of  a talkspurt. 

The  simulations  were  performed  as  follows.  First  the 
delay  string  is  ''randomly”  divided  into  sentences.  At  the 
beginning  of  eac'h  sentence  a selection  of  D is  made  based  on 
the  particular  algorithm  (range  monitor  or  delay  tracker) 
being  simulated.  Knowing  the  delay  of  the  first  message  of 
the  sentence  and  r we  determine  the  number  of  gaps  which 
would  have  occurred  were  this  the  sequence  of  delays  experi- 
enced by  an  actual  sentence.  This  is  repeated  for  each 
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sentence  In  the  strinq  and  the  results  are  recorded. 

iie  show  only  the  results  using  two  of  the  Measured 
delay  strings.  is  now  expressed  in  msec  instead  of  a mul~ 
tiple  of  the  mean  delay.  In  Figures  5. 28(a)  and  5.29(8)  are 
the  results  of  the  range  monitoring  techniques  using  » = 2# 
3^  4,  5,  6,  7,  Z,  9/  10/  12/  14/  16/  18/  20/  25/  30/  35/  40/ 
45/  50/  60/  70/  80/  90/  and  100/  with  k = 0/  1/  2/  3/  4/  5/ 
and  10  (k<m-1).  The  performance  retains  the  hyperbolic 
character  to  a large  exte'^t.  Notice  that  a wide  range  of 
m/k  pairs  give  performance  below  6E=.03  (and  6I=.06/  not 
shown).  The  m>40/  k^.OSm  rule  provides  good  performance 
here  as  well/  though  not  apparent  from  the  figure. 

In  Figures  5.28(b)  and  5.29(b)  are  the  results  of  the 
delay  tracker  methods.  The  constant  c scheme  (represented 
by  x's  in  the  figures/  each  x is  the  result  of  a single 
value  or  c)  performs  quite  well  considering  its  simplicity. 
One  must/  however/  be  quite  careful  in  the  selection  of  c. 
Note  that  the  range  of  "good"  c varies  with  delay  distribu- 
tion/ which  is  effected  by  a number  of  factors  of  which  the 
user  is  unaware  (i.e./  network  distance/  network  load/ 


S^igura  S.28(a).  Ranga  pradiction  parformanca  tinwlation  (Method  E,  inaaturatnant,  1 hop). 
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Fifura  5.28(b).  May  trackar  parformanca  tiinulation  (Mathod  E,  maaturamcnt,  1 hop). 
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A very  interesting  fact  is  now  apparent  regarding  the 
performance  of  the  scheme  which  dynamically  selects  c.  Each 
point  (a  ♦ sign  in  the  figures)  shows  the  performance  for  a 
particular  initial  value  of  c.  Larger  values  of  c may  cause 
both  larger  mean  delay  and  more  gaps!  while  the  performance 
is  quite  good  in  terms  of  gap  probability/  fi  depends  criti- 
cally on  the  selection  of  the  initial  value  of  c.  This  ins- 
tability is  effected  by  the  selection  of  the  gapless  string 
length  parameter  s.  It  would  probably  be  possible  to  find 
an  s which  would  result  in  stable  behavior  for  these  partic- 
ular delay  strings.  Finding  an  s which  guarantees  stability 
for  all  (or  even  some  most  likely)  strings/  however/  is 
probably  impossible. 

3.4  CQQ£iusiQQ£ 

Regarding  source  buffering  for  stream  traffic  communi- 
cation/ we  found  that  no  packet-by-packet  strategy  could 
adapt  to  the  rapidly  changing  network  delay  in  a way  which 
effectively  reduced  end-to-end  delay  (or  its  variance)  or 
significantly  increased  throughput.  Therefore/  we  conclude 
that  sending  strategy  should  remain  fixed  for  long  periods 
and  only  adapt  to  major  changes  in  network  performance/ 
perhaps  by  using  feedback  from  the  destination  buffering 
algorithm . 

In  the  area  of  destination  buffering/  the  results  are 


more  promising.  One  can  devise  buffering  schemes  which 
effectively  balance  the  frequency  (and  duration)  of  output 
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gaps  against  end-to-end  delay.  Ue  have  produced  a framework 
by  which  such  schemes  may  be  compared.  Ry  applying  that 
framework/  we  have  found/  for  example/  that  among  these' 
techniques/  delay  range  prediction  is  advantageous  to  delay 
tracking.  This  is  the  case  because  delay  tracking  requires 
user  input  and  may  be  unstable  in  certain  cases.  Our  ap- 
proach of  looking  at  each  end  of  a continuum  of  playback 
schemes  has  produced  bounds  of  performance  within  which  the 
performance  of  all  schemes  in  the  continuum  must  lie.  We 
have  gained  in  understanding  the  tradeoff  between  gaps  and 
delay.  The  above  should  provide  some  useful  tools  for 
designing  new  and  better  buffering  and  playback  schemes. 

In  a more  general  sense  we  have  gained  some  insight 
into  the  area  of  network  delay  distribution  and  the  indepen- 
dence of  successive  delay. 


CHAPTER  6 


CONCLUSIONS  AND  SUGGESTIONS  FOR  FUTURE  RESEARCH 


6.1  CfiQ£iusl2ns 

we  have  identified  three  areas  within  the  ARPANET  cro- 
C6  ures  in  which  the  performance  of  stream  traffic  communi- 
cation may  be  significantly  enhanced.  In  each  case  it  was 
demonstrated  (by  simulation)  that  the  traditional  forms  of 
data  communication  (i.e.^  the  LO  and  HT  classes  defined  in 
Chapter  1)  would  have  improved  performance  as  well.  The 
underlying  cause  of  each  of  the  areas  of  poor  performance 
was  eventually  found  to  be  the  routing  update  procedure. 
However/  the  lessons  learned  are  worth  separate  mention.  We 
now  state  three  rules  of  thumb/  derived  from  this  research/ 
which  apply  in  general  to  packet  switching  (at  least  of  the 
ARPANET  variety). 

1)  Routing  loops  must  be  controlled. 

3)  Great  care  must  be  taken  whenever  a locally  optimum 
policy  is  violated/  in  order  to  insure  that  the  vio- 
lation improves  global  performance  in  some  signifi- 
cant way. 

3)  In  (at  least)  large  networks/  periodic  update  rout- 
ing procedures  require  excessive  overhead/  There- 
fore/ for  large  networks  we  must  update  routing 
information  by  schemes  which  are  asynchronously 
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triggered  by  the  failure  or  repair  of  network  com- 
ponents or  by  heavy  congestion  in  the  network. 

In  Chapter  2 we  found  that  the  most  effective  periodic 
update  routing  procedure  (of  those  considered)  was  the  local 
loop-free  scheme  LLFR.  The  general  loop-free  scheme  LFR  was 
less  effective  because  of  the  infrequent  occurrence  of  mul- 
tinode loops  coupled  with  the  increased  overhead  required  to 
prevent  multinode  loops.  However#  it  should  be  clear  that 
under  an  asynchronous  update  procedure  we  must  require  each 
update  to  produce  a loop-free  set  of  routing  tables.  There- 
fore LFR  is  a candidate  for  a distributed  asynchronous  up- 
date routing  procedure  in  large  networks.  We  could  call 
this  new  technique  ALFR  (pronounced  as  a New  Fnglander  pro- 
nounces the  first  letter  of  the  of  the  Greek  alphabet). 

Rule  2 is  generated  by  our  discovery#  in  Chapter  3# 
that  processing  routing  packets  at  low  priority  produced 
both  local  and  global  performance  improvement  in  terms  of 
average  delay.  Under  an  asynchronous  update  scheme  this 
discovery  may  not  have  been  made.  Perhaps  under  such  a 
scheme#  since  updates  are  triggered  by  something  very  "bad” 
(or  good)#  high  priority  routing  may  produce  better  perfor- 
mance globally  even  though  it  would  still  be  worse  locally. 

We  found  in  Chapter  4 that  periodic  update  routing  is 
costly  in  large  networks.  With  periodic  update  routing#  in 
order  to  insure  that  news  of  a failure  is  propagated  through 
a large  network  it  is  necessary  to  reduce  the  period  of  the 
routing  updates.  As  the  network  grows  bigger  so  does  the 


overhead  due  to  routing.  With  such  high  overhead  the  pui — 
pose  of  adaptive  routing  (i.e.»  to  reduce  average  delay  and 
increase  throughput  in  the  face  of  changing  topology  or 
traffic  patterns)  is  defeated. 

Our  investigation  of  sending  and  receiving  strategies 
has  yielded  the  following  results.  We  found  no  effective 
method  of  using  delay  prediction  on  a message-by-message 
basis  to  assist  in  the  sending  strategy  due  to  the  large 
variability  of  delay  for  successive  messages.  Therefore^  we 
conclude  that  the  sending  strategy  should  remain  fixed  for 
some  “long”  periods  of  time  and  modified  only  as  a result  of 
large  variations  in  network  performance.  The  study  of  des- 
tination buffering  proved  to  be  more  fruitful.  The  delay 
tracking  methods  are  attractive  because  of  implementation 
ease/  but  the  requirement  that  the  user  specify  a starting 
value  is  undesirable.  For  "good"  starting  values  the  delay 
trackers  perform  as  well  as  the  slightly  more  complicated 
range  predictors.  Unfortunately/  however/  a good  selection 
of  starting  value  is  dependent  on  factors  of  which  the  user 
is  completely  unaware.  On  the  other  hand/  using  a fixed  (or 
automatically  adjusted)  m-k  pair  allows  the  automatic  selec- 
tion of  "destination  wait  time"  based  more  on  the  network 
dynamics.  Therefore  range  prediction  is  preferable  to  delay 
track  ing. 
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6.2  iiuaa£&liaD&  fat  fufute  iQ^£itia§flaa 

Several  areas  related  to  this  work  have  been  left  un- 
touched here.  We  have  considered  only  part  of  the  larger 
problem  of  network  design.  That  is^  to  optimize  the  design 
(e.g.^  minimize  the  cost)  of  a network  which  is  to  carry 
several  classes  of  traffic^  each  with  a different  set  of 
constraints^  over  the  set  of  design  variables  which  includes 
topology/  capacity  assignment/  routing  procedure/  system 
priorities/  buffering/  and  flow  and  congestion  control. 
Some  of  the  more  fruitful  individual  areas  would  seem  to  be 
routing  and  flow  and  congestion  control.  We  stated  earlier 
that  asynchronous  update  routing  procedures  deserve  investi- 
gation. Some  work  has  been  done  by  Kleinrock  and  Fultz 
CFult  72D  in  this  area  for  small  networks.  The  topic  should 
be  further  studied  especially  with  regard  to  large  networks. 

In  Chapter  5 we  modeled  the  stream  traffic  delay  as  an 
independent  random  variable  possessing  some  probability  dis- 
tribution function.  A better  characterization  using  random 
processes  would  allow  the  creation  of  more  accurate  models 
of  the  destination  buffering  problem  as  well  as  aid  in  the 
above  design  problem.  The  buffering  problem  itself  could 
also  be  extended  to  include  a notion  of  gap  duration.  The 
model  should  also  be  used  or  extended  to  provide  information 
regarding  the  minimum  buffer  size  required  (at  the  destina- 
tion device)  to  guarantee  certain  performance. 

Finally  the  buffering  analysis  could  be  extended  im- 
mediately to  the  other  packet  technologies  mentioned  in 
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Chapter  1 by  providing  a delay  distribution  in  each  case. 
Also#  the  destination  buffering  and  associated  delay  moni- 
toring could  be  used  to  provide  feedback  to  the  source  in 
order  that  changes  be  made  in  sending  strategy  and  the 
compression  scheme#  at  least  in  the  case  of  speech. 
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